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PREFACE 



i 



The conierence, Evaluation in Continuing Medical Education, was presented by the 
Department of Postgraduate Medical Education of the University of Kansas School of 
Medicine with the support of the Bureau of Health Professions, Education and Manpower 
Training of the Public Health Service, Department of Health, Education, and Welfare 
on August 25 and 26, 1970. The sole objective of the conference was to assemble ex- 
perts in various fields of continuing education for health — different types of 
producers of various kinds of courses or programs, adult educationalists, professior.- 
al evaluators, financers of programs, etc, -- to discuss the evaluation of euch programs. 
We wanted to give particular attention to defining the obstacles or problems encountered 
in evaluation and outline possible solutions to them with the goal of recording their 
deliberations for the benefit of others — particularly newcomers to the field — who 
might: desire to know what those who had been in the ''business^' longer had to say. 

There is no doubt that most of those in attendance learned something (or at least 
had their perspectives deepened) , but that was not the objective of the conference. It 
was rather to indicate the complexities of the field of evaluation of continuing medical 
education efforts and to help those with less experience or with a more restricted point 
of view. For that reason we made no effort to arrive at any ’'tk solutions , conclusions, 
or formal recommendations. We have tried instead to record th? extemporaneous proceedings 
with maximum faithfulness. Editorial privilege has been used sparingly, being applied 
only to eliminate irrelevant side remarks (and some very good jokes) and to try to be 
sure that the meaning of the spoken word was transmitted faithfully on the rare occasions 
when oral discourse, although clear in itself, lost its meaning on the printed page. 

Logistical problems made it difficult to submit a transcript of each discussant’s 
comments to him for approval, and I hope that all of them will pardon me for taking 
the liberty of presenting their remarks as they were transcribed by the stenotypist, 

Ti;e Department of Postgraduate Medical Education has recently had critiques of many 
of its programs by Ethel Nurge, Ph.D,, cultural anthropologist. Unfortunately Dr, 

Nurge was unable to attend the conference, but was kind enough to read the proceedings 
and comment on them. Her comments are included in Appendix A. 

It is obvious that this conference could not have been carried out successfully 
without the maximum cooperation of all those who attended, and I want particularly to 
thank the Consultant Planning Group, the Conference Consultants, and the volunteer Group 
Discussion Reporters for their roles in planning and producing the conference. It is 
also a pleasure to thank our extremely capable reporter, Donna L, Dunwoody, Dunwoody 
Convention Reporting, Shawnee Mission, Kansas, who produced a truly superb transcript 
of the extemporaneous proceedings -- a transcript that made the job of editing a 



pleasure 



Jesse D. Rising, M,D 
Kansas City, Kansas 
October, 1970 
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FIRST PLENARY SESSION 
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EDUCATOR ATTITUDES TO EVALUATION 



2 



Clement R. Brown^ MrD. 



To better serve the medical educators with whom I have consulted during the past 
two years, I have administered a series of instruments to measure their knowledge, skills 
and opinions relative to continuing medical education program design. I will report 
briefly or responses from an opinion survey. 

I . Current Educator Attitudes to Evaluation of Continuing Medical Education (CME) 

I administered the opinion survey to 157 people directly involved with continuing 
medical education program production. Most were physicians. Most who were nonphysicians 
had a background in the behavioral sciences. A few were medical record librarians. In- 
cluded in the group were members of medical school departments of continuing medical 
education, regional medical programs, and directors of medical education in community 
hospitals . 

The opinion survey consisted of a series of statements concerning continuing medical 
education with instructions to the respondents to select those statements with which they 
were in greatest agreement or disagreement. Statements 2, 16 and 17 are most pertinent 
to the present discussion. Statement No. 2 — "Success of CIE programs must be measured 
in terms of improved patient care." Drr-hundred forty-nine OE program producers responded 
to this statement — 135 or 90 percent agreeing, and only lA or 10 percent disagreeing 
with the statement. 

Statement No. 16 — "It is possible to measure the effect of CbE programs on patient 
care, and also to determine the need for CbE programs by measuring patient care." One- 
hundred thirty-three responded to this statement -- 118 or 89 percent agreeing, and only 
15 or 11 percent disagreeing. 

Statement No. 17 — "The true success of a Cbffi program can best be judged by a 
"before*^ and "after'’ examination (knowledge test) of the participants." One-hundred 
thirty-five responded to this statement -- only 17 or 12 percent agreed, while 118 or 
88 percent disagreed with the statement. 

Thus, most of the respondents agre'^d that CIE programs can and must be evaluated 
in terms of their effect in improving patient care rather than by pre- and postprogram 
paper and pencil exams. 

Three further statements on this opinion survey seem worthy of mention. Statement 
No. 20 — "Much valuable physician time is wasted in attending current standard OE 
programs." One-hundred twenty-five CIE program producers responded to this statement -- 
56 or 45 percent agreein2 , and 69 or 55 percent disagreeing. Such a response of OE 
program producers implies a necessity for change, although I recognize the word "standard" 
may be open to various interpretations. 

Statement No. 13 — "Behavioral scientists have little to offer physician educators 
in the design of CME programs," Of 1*3 responses only 15 or 10 percent were in agree- 
ment; 128 or 90 percent disagreed. Increased use of behavioral scientists in design of 
continuing medical education programs seems indicated by this response. 

Finally, Statement No, 15 -- "Incentives like the ’Physician's Recognition Award' 
of the AMA are likely to have a strong motivating influence in securing physicians' CME 
program attendance." of 122 responses only 14 or 11 percent were in agreement, while 
108 or 89 percent disagreed. 
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\>Jhat do these responses mean? In my estimation^ they represent a necessary but in- 
sufficient condition for progress. As opinions, they represent intellectual agreement. 
But we are looking for evidence of evaluation activity on the part of CME program pro- 
ducers, Favorable attitudes should be reflected in educator behavior. Behavioral 
evidence of educator attitudes favorable to program evaluation, particularly that 
which measures its effect on patient care, seems strikingly absent. What is the problem? 
Unfortunately, I don’t have data on educator attitudes toward evaluation except as can be 
implied or assumed from the almost complete lack of evaluation of current CME programs, 
particularly as we move from simple head counts as a method of evaluation, through paper 
and pencil exams, to the effect c: CME programs on patient care process, to the final 
criterion of CME success which is improved patient care and results. For the purposes of 
this conference may we accept the seemingly obvious fact that there are overwhelmingly 
negative attitudes toward evaluation of continuing medical education programs by theii 
producers, particularly as we move toward more meaningful types of evaluation. And let 
us accept as the challenge of this conference the improvement of educator attitudes 
toward evaluation of continuing medical education. 

II. How to Improve Educator Attitudes Toward Evaluation of Continuing 

Me ‘ i cal Education 

To Improve educator attitudes toward evalusMon of continuing medical education 
programs or, more precisely, to Inctease the number of educators who are evaluating the 
effect of their CME programs in Improving patient care, we need to do three things: 
a) We must improve the conditions surrounding evaluation, b) we must improve the con- 
sequences following evaluation, and c) we must provide models; that is, some highly 
respected producers of continuing me;dical education programs who ^re evf.luating the 
effect of their programs in terms of improvement of patient care. 

Posi t ive Conditions Surrounding Evaluation 

Here is a list of conditions that might exist during the e\aluation process that 
would tend to i ncrease evaluating activity (action, behavior) on the part of CME program 
producers . 

1. The CME producer can demonstrate to himsei.f and to the learner that the learner is 
progressing (learning) . 

2. Learner recognizes producer’s Interest in him in providing evidence of his success. 

3. Learner recognizes that evaluation of education in terms of its effect on his 
practice keeps education relevant to his needs, his patient care problems, his 
practice. 

4. Feedback to learner from evaluative process enhances learning where learner behavior 
is appropriate (the principle of programed learning) . 

5. Inappro,. riate learner behavior is not supported, making ^learning (a very difficult 
process) unnecessary, 

6. Saves producer and learner time — don't teach what learner knows. 

7. Makes more individualized teaching possible, 

8. Game atmosphere that can exist during the evaluative process can be enjoyable. 

9. Increased grant support usually obtained for programs that are well evaluated, 
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Aversive Conditions Surrounding Evaluation 
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Here is a list of condlt^.ons that might exist during the evaluation process that 
would tend to decrease evaluating activity (action, behavior) on the part of CME program 
producers . 

1. Demonstration of lack of program success. 

2. Demonstration of inappropriateness of learning experiences to achieve agreed upon 
educational objectives. 

3. Producer frustration in achieving skills necessary to evaluate. 

4. Evaluation can be expensive. 

5. ^'Loss*' of producer and learner time by participating in the evaluative process, 

6. Learner fear of producing evidence of lack of achievement, whether evidence is from 
exams, observation of practice skills, chart review, or any measures that show lack 
of improvement of patient care process or end results. 

7. Learner fear of evaluative process as a carryover from medical school where certifying 
function of exams may have been stressed rather than their effect of enhancing 
learning through feedback. 

Positive Consequences Following Evaluation 

Here is a list of consequences that may follow evaluation activity that would tend 
to increase further evaluating activity (action, behavior) on the part of CME program 
producers . 

1, Educator has solid evidence that the education program was a success patient care 
improved . 

2, If the desired behavioral results were achieved it may indicate- that the educational 
techniques, learning experiences, were appropriate to the objective, 

3, Evidence of success may Improve subsequent physician attendance and participation. 

4, Improved patient care directly related to education program activity provides the 
more powerfully motivating force of an intrinsic incentive, 

5, Learning time may be decreased with feedback from the evaluative process, 

6, Evidence of success may bring additional financial support. 

7, Success in terms of improved patient care may Justify the CME producers' existence — 
in fact, what else should? 



Aversive Consequences Following Evaluation 

Here is a list of consequences that may follow evaluation activity that would tend 
to decrease further evaluating activity (action, behavior) on the part of CME program 
producers , 

1, Educator may secure evidence that no behavioral change occurred in the physician; 
patient care was not improved; the program was a failure, 

2, Failure of the program as documented by evaluation activity may indicate the learning 
O sriences, the education techniques, were inappropriate to the educational ob,*- 

ERJC^ives. 

ir> 
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Lack of demonstrable success may decrease attendance. 
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4. Lack of learner achievement may discourage the learner. 

5. Lack of relation of education program activity to patient care Improvement may be 
demonstrated to the learner. 

6. Failure may require that the educator learn something about education. 

7. '‘Loss'^ of learner and educator time In evaluative activity. 

8. Evidence of failure may reduce financial support. 

Modeling 

Finally, to increase the Incidence of evaluative behavior on the part of CME 
program producers, we must have acceptable models of program evaluation exhibited by 
educators who command respect. Herein lies the second challenge to those present. If 
most of you begin evaluating your CME, you could provide the models and modeling 
necessary. Very possibly you have this responsibility to those In the field of CME 
who accept you as their leaders. Ke feel that we have an acceptable model at Chestnut 
Hill Hospital, at least for community hospitals, and through workshops I will be con- 
ducting at the Universities of Illinois, Colorado, Washington! and In California, will 
be attempting a replication of our model. We need acceptable models at othe^r levels; 
l.e., medical school CME programs and RMP's. 

Acceptable models of CME program evaluation will be those that exhibit most of the 
positive conditions and consequences surrounding and following evaluative activity and 
few of the aversive conditions and consequences. 

Reference 
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Developing Attitudes Toward Learning . Palo Alto; F(»arson Publ., 
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Select those statements with which you are in greatest agreement and place the number 

that preceeds them In these boxes. 

( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) { ) ( ) 

1. CME directors usually know the course content needed for physician participants. 

2. Success of CME programs must be measured in terms of improved patient care. 

3. The central problem in CME program development is motivating the practicing phy- 
sician to attend. 

4. In the design of CME programo, the patient's needs should be met first by defining 
areas of poor patient care. 

5. A CME program is best held in a location where the practicing physician can get 
away from his practice and its problems. 

6. If the care actually provided to a patient population can be compared to an ideal 
or criterion physician performance, the physician behavior needed to transform 
actual care to ideal care can be translated directly Into education objectives. 

7. Most practicing physicians have little desire to keep up with the advances in 
medical practice. 

8. A system of priorities seems essential in determining which patient care needs 
shall be met first. 

9. It is a waste of time to ask physicians what they want before designing CME 
programs . 

10. In deciding where to start a CME program, one should determine the greatest causes 
of preventable disability and select those where physicians are doing little to 
prevent disability. 

11, Behavioral scientists have little to offer physician educators in the designs of 
CME programs. 

J2, It will be absolutely essential to educate substantial nurabtra of allied health 

personnel to do many of the things physicians are now doing if we are to meet the 
health care needs of the country. 

13, Incentives like the "Physician* a Recognition Award" of the AMA are likely to have 
a strong motivating influence in securing physicians* CME program attendance. 

14, It is possible to measure the effect of CME programs on patient care, and also 
tc determine the need for CME programs by measuring patient care. 

15. The true success of a CME program can best be Judged by a fore and post-examination 
(knowledge test) the participants. 

16. Much valuable physician time Is wasted in attending current standard CME programs. 

Now select those statements above with which you are in greatest diaagreement and place 

the number that precedes them in these boxes. 

( )^ i } ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) 
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Now please respond to these statements. 
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17. 1 have evaluated at least one of the CME programs I have conducted by measuring 

its effect on patient care. Yes No 

18, I have developed the curriculum for at least one of the C^^E programs I have 
conducted by first gathering data to substantiate patient care deficit (s) , 

Yes No 

19, I have gathered data representing patient care deficits before and after at 

least one of the CME programs I have conducted and found significant improve- 
ment in patient care after my CME program. Yes No 

20. I have been unable to ove:come the obstacles present at my hospital to center my 

CME programs on documented patient care deficits and to measure effects of my 
program on these deficits. Yes No 

21, I feel that my medical staff and/or I do not have the skills necessary to center 

our CME program on patient care needs (deficits) . Yes No 

22. My medical staff and I would like to achieve the skills necessary to center our 

CME programs on patient care needs (deficits) , Yes No 
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PHYSICIAN ATTITUDES TO EVALUATION S^ 
Neal A. Vanselow, K.D. 



In spice of recent emphasis placed upon evaluating the effectiveness of continuing 
medical education programs, it is surprising that a search of the literature reveals 
little objective data on the attitude of the practicing physician toward such procedures. 
In preparing this paper, a concerted effort, including a IfEDLARS search, failed to reveal 
much more than a few casual references to this topic. For this reason, the conclusions 
arrived at here are based as much upon opinion as they are on fact. A prospective study 
aimed at determining physician attitudes toward the various types of evaluation schemes 
now In use or proposed for the future seems highly desirable before large-scale evalui- 
tion efforts are undertaken. 

Three basic assumptions are made in discussing this topic. The first of these Is 
that doctors in practice have had little training in educational theory and educational 
techniques and are therefore not inherently attuned to the crucial importance of evalu- 
ating the effectiveness of an educational undertaking. Unfortunately, the only evaluation 
most physicians have been exposed to In their undergraduate and graduate training programs 
is evaluation aimed at testing the knowledge and skills of the student himself. In this 
atmosphere it was usually assumed that any failure was due to a defect in the student, 
and never to a defect in the educational experience. Because of this It appears obvious 
that those who are Interested in evaluating continuing medical education programs have 
a selling job to do. The practicing physician must be convinced that the time and effort 
he contributes to evaluation will ultimately be of benefit to him. 

A second assumption proposes that selling evaluation will not be as difficult as 
some would have us believe, and for a very important reason — the practicing physician 
is genuinely Interested in the quality of medical care he delivers. If he can be con- 
vinced that evaluation will ultimately result In Improved care for his patients, he will 
be willing to take the time and effort to cooperate. It hardly seems necessary to defend 
this assumption, but for those who wish to challenge It, the following data are offered. 

In 1969, the Center for Research on Utilization of Scientific Knowledge at the University 
of Michigan, in cooperation with the Department of Postgraduate Medicine of the Univer- 
sity of Michigan Medical School, conducted a survey a stratified fandom sample of 
physicians practicing in Michigan, This Michigan Physic.'^ ans ^ Survey Included 1,600 of 
the 8,000 Doctors of Medicine and Doctors of Osteopathy In the state, and was designed 
to learn more about their attitudes toward continuing medical education. Usable data 
was collected from 1,100 physicians (70% of those surveyed). As part of the survey, 
each participant was asked to indicate the degree to which he was bothered or worried 
by a number of factors. Table 1 illustrates that, of the ^'top ten worlas" of physicians 
In Michigan, the first is concern for the quality of work they are doing. In addition, 
two of the top three factors which bother phyaiclans in practice relate directly to the 
matter of quality medical care. 

The final assumption, If valid, may exert an adverse effect on physician acceptance 
rf evaluation. In recent years there hss been Increasing pressure fro^ | variety of 
sources for mandatory re-certlf ication or re-licensure of physicians, * Data from 
the Michigan Physicians' Survey (Table 2) Illustrates that a majority of doctors In 
practice are concerned about this trend. Approximately half of the participants re- 
sponded in the two most negative categories when asked to indicate the extent to which 
they favored periodic re-examitiatlon as a requirement for llsccnsure. Only 15 percent 
of the physicians surveyed responded In the two most positive categories- This data is 
of importance to the medical educator since many of the techniques now used to evaluate 
the effectiveness of a postgraduate course can also be used for policing or punitive 
purposes. For example, direct testing of a physician's knowledge or a medical audit, 
designed to measure the level of patient care he delivers, could also provide data 
for re-ll:censure or re-certification. It appears valid to assume that recent pressure 
for mandatory re-evaluatlon of a physician's competence will exert a negative Influence 
receptivity ^‘ ^ward meaningful evaluation of continuing medical education. The 
CDIf^Ted by Neal A. Vanselow, M.D. jnd Will Ian L- Schmalgemel er , M.A. 



butden will be on the educator to convince his physician-student that the data he collects 
will be used only for evaluation purposes and will not be released to agencies concerned 
with policing the profession. 

Present Evidence on Physician Attitudes — Fact and Opinion 

Four major evaluation techniques have been used in recent years. They are; the 
’’head count”; the "popularity contest"; measurement of factual knowledge by pretesting 
and posttesting; and measurement of changes in physician behavior as related to patient 
care In the order listed, each technique is generally conceded to be more satisfactory 
and sophisticated than the preceding one. 

The "head count” technique has been used for years on the rather naive assumption 
that the effectiveness of a postgraduate program can be measured adequately by counting 
the number of physicians who attend it. Because of the obvious fallacies in this line 
of reasoning., as veil as the fact that it involves no effort on the part of the attendee, 
no comment need be made regarding its acceptability to the practicing physician. 

The ’’Popularity contest” is the time-honored technique of asking each enrollee for 
his subjective Impressions of each presentation and the educational program as a whole. 
This techniqv' has been used In our courses at ttie University of Michigan for years. The 
evaluation form we use, shown in Figure 1, is probably no better and no worse than forms 
used elsewhere, and is presented merely as an example of this type of evaluation. In 
Gur experience, such evaluation efforts have been well accepted by the physician in 
practice. The course director must continually remind the course enrollees to fill out 
the forms but, if such reminders are given, a high yield of completed evaluations is ob- 
tained. The resultant comments are usually candid and helpful in planning future 
activities. We do not require the physician to put his name on the evaluation form. 

The resultant anonymity has probably been helpful, since the evaluation form is in no 
way threatening to the individual completing it. 

An int eresting^variation of the ’’popularity contest” approach has been used by 
Dr Thomas C. Meyer at the University of Wisconsin Medical School. Prior to a course, a 
small number of enrollees is selected randomly and asked to meet with the course 
director. They are queried regarding their expectations, and are also requested to 
meet with the course director at the end of each session to critique the presentations 
given that day. Dr, Meyer has found this technique exceedingly useful. He has observed 
that the enrollees will give pertinent and candid comments, even when their critiques 
ate taperecorded . In his experience, this type of evaluation technique has been eagerly 
and enthusiastically accepted by the practicing physician. 

Although the evidence suggests that the "popularity contest" approach to evaluation 
is well accepted by physicians in practice, the evidence for acceptability is lass clear 
for techniques which include testing of factual knowledge. In the latter, each physiclan- 
scudent is tested on course content before and at varying periods after the course. 

Short objective tests of the multiple choice variety are commonly used. Thi*> method of 
evaluation is potentially more threatening to the practicing physician, since it permits 
a third party to assess directly what he knows and what he does not know. In spite of 
this. Dr. Stephen Abrahamson has stated that, in his experience, short objective tests 
administered at t\»^ beginning^of a program and at its completion seem to be routinely 
a\.cepted by the participants. 

Two other bits of evidence would seem to confirm this view. The first of ^hese is 
experience with the Self-Assessment Test of the American College of Physicians. De- 
veloped by the College's Committee on Edu.:ational Activities, this test permits practicing 
internists to test their knowledge in nine areas of internal medicine by voluntarily 
agreeing to take » /OO-question multiple-choice examination at hone. Initially, 5,000 
of the 15,000 ACP member'^ took the test. Since the initial offering, gn^agditional 
5,600 members and norunembers have participated. From all indications, * * the program 
has been enthusiastically received, and the American Psychiatric Association and American 
' » of Pediatrics are planning simlla^r undertakings. Caution must be used however 
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when attempting to generalize from this experience, The Self-Assessment Test was accepted 
by practicing internists, but great care was taken to minimize its potentially threatening 
aspects. It was purely voluntary, A bonded firm (separate trom the College) handled 
the applications, mailed the test papers, and scoied the test results. Participating 
physicians could be identified only by coded numbers. Only the participating physician 
had access to his performance. Because of the fear chat the data might be improperly 
used, no group scores were compiled in spite of the fact chat such data might have been 
valuable in directing future continuing education efforts of the College, Under these 
circumstances, internists cooperated; what might have happened if confidentiality had 
not been guarded so closely is a matter for speculation. 

The response to th^^ Individual Physician Profile project at the University of 
Wisconsin also suggests that physicians will permit testing of their factual knowledge 
when done in a d^sciete and ncnthreateniug way,^ Developed by Dr. Thomas C, Meyer and 
associates, this program was initiated in 1968, Medical secretaries visited Che office 
of each participating physician and recorded data on the tyoe of patient seen In the 
office, at home, and in the hospital. Based upon his practice profile, a multiple - 
choice examination was constructed for each physician, testing his knowledge in those 
areas of medicine which were commonly encountered in his practice. From his practice 
profile and performance on the examination, a program of continuing medical education 
was then individually tailored for each participant, Thirty-seven physicians participated 
in the first year of t!vj study and responded enthusiastically. Only one of the original 
37 declined to participate a second year. The original experience may be misleading how»- 
ever , since most of those participating were friends or personal acquaintances of Dr. 
Meyer. In addition the original group w^as a high'y motivated one, already participating 
in continuing medicaJ. education and considered to be secure in the quality of medical 
care they were providing. Of more significance is the fact that 60 physicians are 
participating in the second year of the project rnd that the number of volunteers has 
exceeded the number of places available. While this project Is an experimental one and 
is not strictly analogous to the situation of testing knowledge for evaluation purposes, 
it does suggest that physicians are 'shilling to submit to tests of factual knowledge u^hen 
used for what they perceive to be constructive purposes. 

The most meaningful type of evaluation is felt to be measurement of changes in 
physician behavior which occur following an educational program. Such measurements can 
be made most easily in a hospital setting and most commonly utilize the medical audit 
as a measurement device. The classical studies of W1 lllamso^^et al . at Roc ford 
Hospital and the vrell-known program of Brown and associates at Chestnut Hill Hospital 
in Philadelphia are excellent examples of what can be accomplished when the medical audit 
is used tc 'esign and evaluate programs of continuing medical education. Wliile the rnedi- 
cal audit was accepted by the s*^affs of both hospitals these two studies are somewhat 
unique, since each was initiated by an outstanding and persuasive inoividual. One could 
argue that each succeeded largely because of the personal charisma of the individual in 
charge, and thc.t it would be dangerous to assume equal success could be anticipated in 
other settings. 

The Medical Audit Program (MAP) of the Commission on Professional and Hospital 
Activities (CPHA) is used by over 1,300 community hospitals in th^^United States and 
several foreign countries. The director of CPHA, Dr. Vergil Slee has found that ac- 
ceptance of the medical audit is dependent in large part upon the individual who directs 
it in the hospital Involved. in Ms experience, some physicians have committed ’’profes- 
sional suicide” because they failed to take Into consideration the worries of the medical 
staff regarding the potential consequences of a program of this type. Dr. Slee feels the 
fear of personal embarrassment is the biggest single factor contributing to physician r_- 
luctance to cooperate. Thn fear of third-party sanctions and the feeling that coercion 
is involved are also given as reasons for opposing the medical audit. Finally, some 
physicians object to the technique Itself. ”A11 you are measuring is medical work,” is 
another. However, CPHA nas also found that most medical staffs perceive their original 
r ^ ) be unfounded once MAP is in operation. 
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Under a Regional Medical Program grant, Dr. Joseph Hess at Wayne State University 
Medical School has. recently instituted a program of continuing medical education in 
three small inner- C3 ; Detroit hospitals. A medical audit is used to determine edu- 
cational j^^eeds and evaluate the effectiveness of the programs instituted to meet them. 

Dr. Hess has found each hospital staff willing to accept a medical audit, but emphasizes 
that two conditions must be met. First of all, the audit must be conducted within the 
existing staff structure of the hospital; in bis experience there has been great re- 
luctance to permit an ’’outside agency" to come in to look at medical records. Secondly, 
the audit must be performed and used in a nonthreacening , nonpunltive way. The latter 
theme constantly recurs in discussing this level of evaluation with knowledgeable in- 
dividuals. Its Importance was perhaps best summarized by Bernard Dryer when he wrote: 
"What is required then is a clear-cut and widely understood distinction between education 
and regulation; so that the energy available for learning can be efficiently utilized, 
with each individual physician aware that only he will decide and where he will ap- 

proach a regulatory body for the more formal awards they offer." 

Additional Data on Physician Attitudes 

In an attempt to obtain seme prospective data on physician attitudes toward eval- 
uation, we recently surveyed the opinions of the enrollees in our annual Korthern 
Michigan Summer Course. This course is presented in a resort area each year and .3 
aimed at the family practitioner. Five half-day sessions are devoted to a discussion of 
a broad variety of clinical topics by carefully selected University of Michigan Medical 
School faculty members. In order to give the participants some experience with at least 
one method of evaluation, a short multiple-choice pre-test on the material to be covered 
was administered on the first morning of the course. A posttest of similar format was 
given on ths final day. 

Each participant was then asked to fill out a questionnaire designed to determine 
his attitudes toward a variety of evaluation technics Forty of the 52 full-time 
registrants i.. the course (77 percent) completed the questionnaire. The results of 
this study are summarized in Figures 2-5, 

Figure 2 illustrates that the great majority of physicians rejected the idea that 
it was either unnecessary or impossible to evaluate the effectiveness of a postgraduate 
program. Few felt that counting attendance was an adequate method of evaluation. 
Measurement of subjective reactions to the program and testing of factual knowledge by 
pre- and post-testing were rated by most physicians as appropriate types oi evaluation. 
Less chan half of those responding felt that evaluation of physician behavior via a 
medical audit was appropriate, which indicates that attempts to utilize this technique 
might be met with considerable physician resistance. 

The data In Figure 3 confirms the postulate that practicing physicians are more 
receptive to tests of factual knowledge than they are to evaluation techniques 
employing the medical audit. The negative response to identification of participating 
physicians by name suggests that the extent to which anonymity can be preserved is an 
important factor in determining physician acceptance of evaluation. 

Figure 4 demonstrates that those responding recognized that evaluation of contiri 
uing medical education might ultimately result in improved patient care. The suggestion 
that testing a physician's knowledge was an Irrelevant nuisance was rejected. The im- 
portance of providing each physician with the results of his performance was strongly 
emphasized, A mixed response was obtained to items b, d and f, each of which probed 
for possible concern over perceived "misuse" of evaluation data. 



The question illustrated in Figure 5 was designed to determine how much physician 
anxiety might be generated by evaluation techniques designed to measure factual knowledge 
or physician behavior in delivering patient care. Surprisingly, the response to this 
Item Indicated less anxiety than might have been predicted, and evaluation methods which 
measured behavioral change did not appear to differ significantly from those which 
measured factual knowledge. 

o . 23 
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Each item in the questionnaire was analyzed to determine if age or performance on 
the pre-test and posttest might influence physician response. The responses of physi- 
cians over 45 were no different than those 45 and under; however, as might be expected, 
physicians showing low gain in knowledge from the course tended to be less receptive to 
knowledge testing as a technique of evaluation. 

Summary 



1, There is little published data on practicing physicians' attitudes toward evaluation 
of continuing medical education programs. Prospective studies are needed to provide 
the medical educator with the information he requires to design evaluation techniques 
which will be acceptable to doctors in practice. 

2, Present evidence suggests that practicing physicians consider the "head count'' to 
be a superficial and relatively meaningless evaluation technique, 

3, Surveys of subjective participant reaction to an educational program are regarded 
favorably. 

4, Techniques designed to measure change in factual knowledge are more acceptable than 

techniques such as the medical audit which are designed to measure alterations in 
physician behavior related to patient care. Both are perceived as posing a potential 
threat to the physician in practice, but there is some evidence to indicate that I 

they can be employed successfully, provided: 

a. An effort is made to explain what is being done and the importance of doin;, it. 

b. The physician is provided with feedback on his performance in a constructive and 
nonthreatening manner, 

c. Knowledge or performance testing for purposes of evalr-^ting a continuing medical 
education program is clearly separated from similar testing for punitive or 
regulatory purposes. 
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Michigan Physicians^ Survey 
’’Top Ten Worries” of Practiciig Physicians 



”To what extent are you bothered or worried by 


any of the following 


things?" 




% Responding to 


Same , 


, a Great 


Item 


or a Very Great 


Extent 




M.D. 




D.O. 


How good a job I am doing 


62 




75 


The effective use of ray time 






53 


Feeling that I am not keeping up to date 
Having to do things I don’t want to do 


^3 




56 


^2 




^1 


Having to put up with and tolerate incompetence 


A1 




50 


Being torn by conflicting demands 


33 




31 


Failure to advance professionally 


32 




^0 


Feeling I am "in a rut" professionally 


30 




^1 


Failure to get ahead financially 


28 




35 


Can’t do what is expected of me 


28 




38 




N=919 




H=183 


Tabl 2 2 








Michigan Physicians’ 


Survey 







"To what extent do you favor periodic re-examinaf ion which would be required to maintain 
your license to practice medicine?” 



Percent Responding 


M,D. 


D.O, 


To a very little extent 


33 






)^9 


/ 


To a little extent 


1 j 


13^ 


To some extent 


26 


28 


To a great extent 


11 


10 






> 


To a very great extent 


A 


6 


No response 


10 


8 




N*919 


N“183 
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EVALUATION 

Department of Postgraduate Medicine 

Physiology for the Clinician ; 

A Practical Review of Selected Topics 
February 16-18, 1970 

These evaluations are of great help to our course chairman and faculty in planning 
luture continuing education programs, Please be frank. 

I. Area of Practice (Generalist, Internist, other - please specify): 

lie Graduate of Year 

III. Please evaluate each session of the course; 





General Evaluation 
Excellent Good Poor 


Was it of Practical 
Value to You? 


Comments 


Monday, February 16 

Review of Basic Mechanisms of 

Gastrointestinal Control 












Current Methods of Evaluating 
Liver Function 












Autoimmune Mechanisms In 
Intestinal Disorders 












Pathogenesis of Acute and 
Chronic Pancreatitis 












Respiratory Mechanics 












Pulmonary Acid-Base 
Problems 












Clinical Patterns of Dis- 
ordered Pulmonary Function 












Use of Respirators In 
Clinical Practice 












Tuesday, February 17 
Basic Mechanisms of Renal 
Transport 












Clinical Evaluation of Renal 
Function 












Mechanisms of Uremic 
Manifestations 












The Nature of Renal 
Tubular Acidosis 












Methods and Keasuremento of 
Dialysis 












Normal Physiology of the 
Renln-Anglotensin System 












Control of Aldosterone and 
Sodium Metabolism 












Secondary Aldosteronism 
Without Hypertension 
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Figure 1 (continued) 
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General Evaluation 
Excellent Good Poor 


Was it of Practical 
Value to You? 


CnmrriQ rt t* c 


Film: Aldosterone 








i 


Basic Mechanisms of 
Myocardial Control 












The Pathophysiology of 

Myocardial Failure 

1» In Ischemic Heart Disease 












2. In Valvular Heart Disease 












3. In Congenital Heart 
Disease 












Basic Mechanisms of Periph- 
eral Vascular Control 












The Pathophysiology of 
Occliisive V? ocular Disease 












Televised Demonstration: 
Methods of Graded Exercise 
Testing 












Hemodynamic and Electro- 
cardiographic Responses to 
Exercise 

















IV. The outstanding feature of this course was: 



V. Suggested improvements before this course is offered again: 
VI. Other conunents (use back of sheet if necessary); 
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Practicing Physicians* Reactions to Evaluation Techniques 
Korthern Michigan Summer Course - June, 1970 



a. 


Testing factual knowledge at beginning and end 


Strongly 


V 






Strongly 




of postgraduate course. 


Agree 




4 


3 2 


1 


Disagree 


b. 


Testing factual knowledge three or six months 


Strongly 




V 




Strongly 




after attending course. 


Agree 




4 


3 2 


1 


Disagree 




Responding to test with name for identification. 


Strongly 




V 




Strongly 






Agree 




4 


3 2 


1 


Disagree 


d. 


Medical audit with intent of improving practice. 


Strongly 








Strongly 






Agree 




4 


3 2 


1 


Disagree 


e . 


Test 1/ national specialty or subspecialty 


Strongly 




V 




Strongly 




organlzat ion . 


Agree 




4 


3 2 


1 


Disagree 




N=«0 
















Figure 4 
















Practicing Physicians* Reactions to 


Evaluation 


Concepto 








Northern Michigan Summer Course 


: * June, 


1970 








a. 


Testing a physician's knowledge with feedback 


Strongly 




V 






Strongly 




of performance is worthwhile. 


Agree 


4 




3 2 


1 


Disagree 


b. 


Testing a physician's knowledge is worthwhile 


Strongly 






V 




Strongly 




to others. 


Agree 


4 




3 2 


1 


Disagree 


c. 


Direct testing of a physician’s knowledge is a 


Strongly 






V 




Strongly 




nuisance. 


Agree 


4 




3 2 


1 


Disagree 


d. 


Testing of a physician’s knowledge is a first 


Strongly 






V 




Strongly 




step toward recertification. 


Agree 


4 




3 2 


1 


Disagree 


e. 


Teoting a physician’s knowledge is good for 


ngly 






V 




Strongly 




educational purposes but doesn't help hie 


Agree 


li 




3 2 


1 


Disagree 




patients. 














f . 


M^^'intenance of a licensed physician's 


Strongly 






\T 




Strongly 




competence is his personal concern. 


Agree 


A 




J 2 


1 


Disagree 



N-40 
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Unnecessary Impossible Count Reactions Pre and Evaluate 

to Test to Test Attendance to Lectures Post Test Practice 



Number of Choices 



17 




20 



NORTHERN MICHIGAN SUMMER COURSE-JUNE 1970 




Ot- = N 



18 





30 



NORTHERN MICHIGAN SUMMER COURSE -JUNE 1970 

Physicians' Attitudes to Being Tested 
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Stephen Abraharason, Ph,D, 



The classic paradigm in education (and educational evaluation) has been prea.;hed 
(and practiced) for some years. Attention paid to this paradigm has sharply increased 
over the last several years as “educationists" have become deeply involved in 

medical education and in continuing medical education. Almost a litany has emerged -- ■< 
powerful hymn, the opening line of which (while it may have variations) basically go ^ 
like this: "First, you must define your obj ec t iveo , “ 

The remainder of the paradigm, of course, suggests that only then can one plan the 
learning experiences, the use of audiovisual aids, the necessary measurement procedures, 
and the like. For those of you still unfamiliar with this theme and its variations, 
there are numerous sources to inform infect you: Miller's Teaching and Learning In 

Med lea 1 ^ Schools ; Abrahamson's article in JAJIA , “Evaluation in Continuing Medical Edu- 
cation""*; Hager’s Preparing Instructional Objectives , The term “infection," is used 
because there are interesting signs and symptoms which appear in one who has first been 
"exposed" to this bit of learning (whether he learns it or not). 

The most dangerous reaction to this apparently harmless paradigm is internal and may 
manifest itself in a variety of ways. The person exposed seems to assume that following 
the steps in educational planning and/or educational evaluation is as simple as the state- 
ment of those steps. Perhaps this is a tribute to those Who have "preached the gospel" 
and won new followers, for we "preachers" have deliberately set out to make the processes 
seem logical, necessary, and worth pursuing, I^Jhat we have neglected, for the most part, 
is to warn that this road tc "educational Heaven" is a difficult and deceptive one. It 
was not for some time, for instance, that "preachers" began attempting to practice what 
they preached — at least in the setting of medical education and continuing medical 
education, Rosinski’s early experience at the Medical College of Virginia in 1959 may be 
the first documented and significant involvement of an educationist in the formal state- 
ment of real educational objectives in medical education. And the biggest assistance in 
this area came from Hager in 1962, when he wrote a little programmed text (then called 
"scrambled t^xt") on how to write objectives to guide the preparation of programmed 
instruction. 



Hager's most significant contribution has been said before by many — but never so 
well, apparently, at least to Judge by the impact. Indeed, Wrinkle said it in 1947 in 
a list of "Ciiteric for Educational Objectives," in which he suggested that educational 
objectives must be stated "in terms of the learner's behavior." By this he meant what 
Hager has since promulgated in exquisite detail. Educational objectives are desired 
outcomes — changes in what the learner does (or demonstrates he can do) as a result of 
the learning experience. In fact, "as a result of" may be only my inadvertent addition 
(a Freudian slip, perhaps?) and should read only "af ter " the learning experience. It 
may be only the educational scientist who has the deep interest in such things as cause- 
effect relationships. 



All of this discussion should point out the most glaring Inappropriateness in edu- 
cational objectives — one which hardly ne^ls mentioning, let alone discussion. It is 
now generally accepted *:hat educational objectlv:s should describe what the learners are 
doing (or can demonstrate that they can do) when the Instruction is completed. It is 
inappropriate to include the "reasons" for holding a program; e,g. , to "cover" the new 
biochemistry; to "reach" tne men "in the field" or "in the front lines"; to "demonstrate" 
the new techniques; to "report" the latest "word"; to hold our annual s ymposium . (In 
all too many instances, that last example is the only statement of objectives to be 
found — and to think: it is now declared "Inappropriate.") 



Of course, it is possible to define these desired outcomes at different "levels," 
Jack Thomson has done this in his distinction between "goals" and "objectives," Miller 



associates did thiSgin making a distinction among "ultimate," "intermediate," 
|/^iediate" objectives. Those plaining programs of continuing medical education 
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want to refer to what the physician-participants will learn in the program; they may want 
to refer to how the physician-participants will change after the program; they may want 
to refer to how health-care patterns or health status itself may be affected. Thus, ob- 
jectives may be defined as desired outcomes at several levels. 

1, The physician-participant will acquire knowledge, gain skills, and/or develop at- 
titudes to certain designated or defined standards. 

2, The physician-participant’s performance in certain areas will be "improved" in cer- 
tain designated ways. 

3, The physician-participant's professional practice will be affected in designated 
directions . 

4, Health-care practices in institutions and communities of physician-participants will 
be modified in designated ways, 

5, The health of communities of physician-participants will be significantly bettv^r in 
designated areas. 

"Realistic" objectives, then, can be defined as those which meet this single 
criterion: there should be reasonable expectation of achievement of the objectives as 
a result of the _program . A four-day symposium on early detection of respiratory disease 
through the use of spirometry, diagnostic radiology, and other screening procedures — a 
symposium which includes lab sessions for physician-participants to use these diagnostic 
procedures under supervision and which offers extensive case reviews -- may be expected 
to Improve physician-participants’ skills in this area. The symposium might even be 
expected (reasonably) to bring about significant changes in professional perform.ance and 
practice of physic ian-part iclpc:nts , Tl\e key phrase "reasonable expectation of achieve- 
ment" will be illustrated further later. 



The definition of "appropriate" objectives requires an object to the phrase: "ap- 

propriate to..." In continuing medical education, objectives may be said to be 
"appropriate" when they are appropriate to the health needs of the physician-participants’ 
respective ccmmunltles and/or institutions, to the practice needs of the physician- 
participants, and/or to the educational needs of the profession. In a hospital where 
medical audit has revealed relatively poor management of patients vlio have suffered a 
stroke, it would be appropriate to offer a program designed to help physician-partici- 
pants "Improve" their practice li, this area. In a community in which there has been a 
sharp Increase in the incidence of venereal disease, it would be appropriate to offer 
a program designed to help physician-participants learn about commuaity resources avail- 
able for assiotance to patients in this area. For physicians who "feel" or demonstrate 
certain lack of information (or skill) it would be appropriate to offer a program designed 
to help meet these felt needs. Finally, lest the author be labeled anti-science or 
anti-intellectual, in instances it might be appropriate to offer a progi am designed to 
help physician-participants acquire soma new basic (or basic science) information now 
thought to be relevant and/or important to the practice of medicine. (Note the qualifiers 
In that sentence; they will need clarification and defense later.) 

Perhaps the best way to further distinguish between "realistic" and "unrealistic," 
and between "appropriate" and "inappropriate" is to present come examples. And, as in 
the case of defining "normal" by illustrations of the "abnormal," it might be fruitful 
to describe what this author believes to be inappropriate and/or unrealistic, especially 
since this is a "position" paper. 



An exaggerated illustration, for instance, of inappropriate objectives is drawn 
from an emerging natioi' in Africa, Designing a continuing medical education program 
with the (correctly, even) stated objective of having physician-participants learn skills 
of vector cardiography -- when there are many significant end unmet health needs jiot 
including heart disease — seems to this author to be in?ippropr late .... inappropriate to 
t'-^ needs of the community, inappropriate to an extreme in this regard. Surely, 

Id be better spent attempting to have physician-participants learn in areas which 
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might signif leant iy affect (for the better) the health of their respective communities. 

And our affluence in the United States does not render us immune to the need to apply 
this kind of thinking in planning continuing medical education. Your own experience will 
provide many examples of programs with objectives completely unrelated to the health needs 
of the communities and institutions of participating physicians — even in the. face of 
serious health problems within those institutions and C' ‘jnunities , The w’ork of John 
Williamson, Bob Evans and C?.cm Brown surely substantiates this position. 

Objectives may be inappropriate to the practice needs of the physician-par ticipants . 
When this happens, one may find what the American Heart Association reported jn a now 
fatiiiliar, clasf*ic, and (unfortunately) all-too-of ten disregarded study. Using a pre- 
test, post-test and delayed post-test design, they discovered a significant improvement 
in physician-participants* ability to recognize arrhythmias as a result of an intensive 
training program, only to see those ’'gains'^ disappear after a period of time — during 
v:hich those physician s undoubtedly had no occasion to apply thos e new-f ound skills . 1 hu s , 

one can say that the objectives of the original program were inappropr iate inappro- 
priate to the practice needs of those physicians. 

Another example of this same kjgd of inappropriateness is taken from a study reported 
by Judilynn Foster and Sandra Lass. While their purpose in conducting that study was 
not what is mentioned here, they serendipi tously discovered something germane to this 
paper, and very interesting. They were attempting to evaluate the effects of a videotape 

on the practices of physicians in a hospital in this instance, ordering scrum-sodium 

and serum-potassium determinations and using potassium supplement where indicated with 
patients who were on diuretics. Two ’’matched" hospitals were employed in the study, in 
one of them, records were reviewed prior to and after the presentation of the videotape; 
in the other, similar record review took place at identical times but without the "edu- 
cational treatment" of the videotape. Interestingly enougli, Foster and Lass discovered 
that physicians in Che "experimental" hospital (i-e>, the one In which the videotape was 
shown) were already ’'performing" at a high quality level of practice (with regard to the 
care needed with patients on diuretics) prior to the introduction of the videotape. 

Thus, one can say that the objectives for using thrt program were inappropr iate ... map 
propriate certainly to the practice needs of those physicians. 

The ''saving grace" for most continuing medical education programs — at least with 
regard to appropriateness of objectives -- comes with the inclusion of the modifying 
phrase "to the educational needs of the profession." Any profession has responsibili ly 
for constantly monitoring the quality of servi/e performed and for keeping practitioners 
performing at desired quality levels. Thus, the medi.:al profession, through its agencies 
of continuing medical education, has the responsibility for helping its practicing 
members "keep up" by acquiring new knowledge, by gaining new skills, by reinforcing at- 
titudes of self-ftxaminat ion ; self-education, and self-improvement. Aid, it is the 
professions' responsibility to define what practitioners should kn^ and know how to dj. 
But it is the responsibility of the profession and not that of Individual scientists ot 
of Individual clinicians to do so. The l^te Chester Hyman, Professor of Physiology at 
the University of Southern California, reported on an interesting incident in his pro- 
fessional life. He was asked to participate in a continuing education program and 
present a series of lectures on the topic, "What's New In Physiology Since Yuu Left 
Medical School?" He asked the hundred-odd physician-par ticipa.its to indicate wlien ihev 
had graduated from medical school. The responses ranged from three years before the 
symposium to fifty-three years before it! The point here is that the professio n had 
abdi.cated to one physiologist the determination of what was "appropriate to the edu- 
cational needs of the profession.” 

It is, perhaps, in this last area of appropriateness that the autlior has a 
"position" v’’ ich may really stir up some controversy. Determination of "educational 
needs of the profession" should np^ be left to the discretion of in llvicuals , however 
expert in their own fields. If possible, thesr experts should work with these who are 
planning programs as well as with representatives of those who will be tli3 pl.ysician- 
participant s . In that way, their own intensive studios ^nd highly specialized know’ledge 
,.411 Wo best applied to the community health needs, t' e Instltuti ^nal health-care i.ee(’s, 
individual practitioner’s educational need-. in ca cs wdiore it is irposslMe to 
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so involve the expert (for the many reasons known to all of us), it would seem important 
to do similar kinds of planning with a local — albeit less well qualified — ’'expert'^ and 
indicate to the ''guest expert'^ the objectives, the limitations of desired ’’content 
coverage, and the like,.., as tactfully, of course, as possible. Through this approach^ 
it might be possible to retain ’’control” of both the objectives and the ’’content," 

So far, we have talked about inappropriateness of objectives — to the health needs 
of institution and community, to the practice needs of the physician, to the educational 
needs of the profession- T,et us consider the m;atter, now, of unrealistic objectives. To 
do so, one m;ust also consider the limitations of the program: the duration of the 

teaching-learning experiences, the size of the learning group or groups, the modes of 
instruction available, the setting for learning, the characteristics of the learners, the 
teaching skills of the Instructors, and the like. With these factors we must also con*- 
sider some tr ied-and-true principles of teaching and learning. 

For ic is an unrealistic objective for a one-hour lecture on vector cardiography 
presented to one- hundred physicians to have those physician-par "-icipants gain appreciable 
skill in vector cardiography, as evidenced in their ability to interpret a set of "un- 
knowns.” It exceeds reasonable expectation of achievement (thus, it is unrealistic) 
because one does not gaii: new skills or improve old ones by listening to a lecture (at 
le St, not usually) r It exceeds reasonable expectation of achievement because the amount 
to be learned demands far more than the time allotted. 



It is equally unrealistic to expect physician-participants’ habits to change as the 
result of a lecture or two, however brilliantly delivered. Habits take time to form; they 
take more time to change. Thus, it is unrealistic for a program to list an objective of 
establishing certain new habits without the time and method planned which might at least 
hold some theoretic hope of achievement. Realistically, to change habits would demand at 
least plenty of time, a series of reinforcing experiences in the new behavior area, small- 
group discussion of insecurities engendered during the course of learning, and follow-up 
reinforcement for the new behavior patterns. 

Another instance ot unrealism in objectives involves misjudgment (or lack of con- 
sideration) of the motivation of the learner. To present a videotape to a group of 
casual "drop-in” learners and hypothesize (another way to state an objective) chat they 
will change their professional behavior significantly Is unrealistic. Again, to bring 
about significant change in behavior requires more than a videotape film, or synchronized 
slide-tape presentation. It would not be unrealistic, if those modes were used in the 
ways described, to expect cha..ge in at Itudes or in levels of professional information 
within that topical area. 

Une more example of lack of realism is that of stating objectives of significant 
change in physician-participants despite a full awareness that the program is being held 
"at home" for the participating physicians, and that past experience has sh that less 

than half. of those who are in attendance at the outset are still In attendance toward the 
end, and that most of the physician-participants will be Interrupted by calls from the 
office, the hospital, or the exchange. This is not to say that programs cannot be held 
"at home”; it does suggest, however, that realistic objectives need to be set. 



The Importance of setting realistic objectives may be considered far greater when 
one more factor of appropriateness is introduced. Earlier, appropriateness was limited 
to three major areas: appropriate to health needs of institution and coriimunlty, to 

practice needs of the physlclan-partlclpar.ts , and to educational needs of the profession 
Now It is Important to add one more factor: cost. One must ask the question, "Is this 

objective worth achievement -- at the projected costs in money, personnel tine, learner 
commitment, necessary materials and equipment;" 



A few years ago J. S. Denson and 1 directed a project supported by the United States 
Office of Education, The purpose (another word for objecLive.O was to test the 
feasibility of constructing a plastic-skinned, computer-controlled, anthropometric manl- 
l O e used in training anesthesiology reridents. The project was successful, as 
from reading about it in T Ime , Life , and other education jovirnals, in doing 



exactly that: denonscrating the feasibility of constructing a patient simulator which 

might be used in training health care personnel. Critics (some of them within the 
bureaucracy of the granting agency) expressed disappointment, saying that we had spent 
more than a quarter of a million dollars to train five anesthesiology residents — and 
only in endotracheal intubation. Had our objective been to train health care personnel, 
the criticism would have been justified. Had our objective been to train five anes- 
thesiology residents in endotracheal intubation, the criticism should have been that 
the objective was inappropriate. The objective (when properly stated) might then have 
been appropriate — 1) to the health needs of the community, 2) to the practice needs 
of the physicians, and 3) to the educational needs of the profession — but still in- 
appropriate, in that an fi* “jcdinate amount of money had to be committed to a relatively 
small number of learners co have them master a rather circumscribed skill. For those 
who see an extended period of time committed by each practitioner to a return to the 
teaching hospital in a refresher-residency as the only means of 'Veal’’ continuing medical 
education, this factor of appropriateness looms of critical importance. 

Less starkly, it is possible for large commitments of tine by busy physician- 
instructors and/cr equally busy physician-students to be devoted to a teaching-learning 
exercise, the objective of which is not important enough to warrant such commitment 
and, thus, is inappropriate. 

A "position paper" connotes that the author will assume a position with regard to 
a given subject — usually for purposes of stimulating thought and discussion. .. .prefer- 
ably in that order. In the event that the position is not clear, here are some summary 
statements , 

1. Many continuing education programs are based upon objectives (stated or unstated) 
that are unrealistic, in that there is not a reasonable expectation of their being 
achieved , 

2. The unrealistic quality comes from a lack of consideration of principles of teaching 
and learning in the planning. 

3. Many continuing education programs are based upon objectives (stated or unstated) 
that are inappropriate to the needs of the community, the physicians^ practice or 
the profession’s needs, and/or are inappropriate because of the inordinate costs in- 
volved in light of the gains to be anticipated. 

It is further my contention, however, that many programs are planned — still — 
without objectives. These "objectives," of course, are the most innappropria te of all. 
More than that, with the present fiscal climate, it is frankly unrealistic for programs 
not to have carefully thought-out and equally carefully stated objectives. During the 
last several years, sponsors of continuing medical education efforts have begun to ask 
for evaluative data to answer the question: "Did our money buy anything in the way of 

Improvement?" That question is becoming more of a demand now. The caution is to those 
of us who are planning programs: make sure that objectives are realistic and appropriate. 
It is 3 certainty that we will be held to our "promises." 
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COSTS OF EV\LUAT ION 



WilliaTn T. Herzog, M.P.H. 



The primary intent of this paper is to raise issues related to the costs of evalu- 
ating continuing education efforts. What are the probable costs of various evaluation 
efforts? What general resource allocation issues are facing the field of continuing 
medical education and what role does evaluation play in these issues? 

The text of the paper offers some general interpretation of the increasing interest 
in evaluation and the relevance of cost analysis to this trend. The Appendix estimates 
the cost of different evaluation approaches of a hypothetical short course in order to pro 
vide some basis for comparison. The final paragraph of this Appendix perhaps best states 
the major resource allocation issue in question. 

In the final analysis one cannot escape a basic dilemma cctonon to all human service 
persuasions in our mechanized culture. On one hand, society is asked to finance exten- 
sive educational efforts with very little documentation. On the other, if it tries to 
correct this by research and analysis, it ultimately must face financing extensive eval- 
uative studies with very little documentation as to the value of the information obtained. 
The obvious recourse in the face of these uncertainties is for society to give greater 
attention to the tangible and more measurable options. The only practical way to prevent 
this is to attempt to develop better estimates of program, effectiveness, to argue the 
intrinsic value of the more qualitative and less measurable aspects of human endeavor, 
and to bring the best minds to bear on questioning t^e apparent but deceptive value of 
the more tangible and visible alternatives. If there is any success, high-quality human 
service might at some time attain the political and social appeal new reserved for our 
massive highway systems, space programs, defense capabilitieSj and impressive building 
programs . 



Implications of Program Budgeting 

Although the problem of competing with bricks and mortar is not new, it has taken on 
some new dimensions that are essential to understanding the cost issue. Beginning on 
August 25, 1965, a relatively novel concept was introduced into the federal budget system. 
Alternatively known as program budgeting , PPBS , ( Planning, Programming, and Budgeting 
System ) , and management by objectives , the concept stemmed from management techniques in- 
troduced in the Department of Defense during the early 1960's. The historical roots of 
program budgeting are found in Frederick Taylor’s Scientific Management and are closely 
linked with the operations research approach developed during World War II and subsequent- 
ly applied in numerous industrial settings. The fundamental proposition of program bud- 
geting is that allocatioii of funds for a given program should be directly related to the 
expected accomplishments of that program when viewed in the context of over-all organi- 
zational priorities. This approach to budgeting may sound traditional and unobjection- 
able, but in fact it represents a major change. Even now the single most imporja^t 
determinant of the annual budget allocation is last year's expenditure pattern. * 

The program budgeting system requires that organizational requests for funds be ac- 
companied by the specific objectives to be attained within the budgetary time frame. 

The stated objectives become the guidelines for measurir g program effectiveness. The 
ratio of cost to effectiveness becomes a method for evaluating organizational efficiency 
in order to allocate the limited available funds. Budget reviev» officers are increasingly 
asking organizations to outline' their expected accomplishments, describe the alternative 
approaches available, and specify which approach is most liKely to maximize gain while 
minimizing the cost and risk Involved. The ultimate application of this approach can be 
found in the business setting where the profit margin is an exact statement of the cost 
effectiveness ratio. 



It is not difficult to relate the current pressure for evaluation to this larger trend 
* ' ness and government management. At a recent conference on medical continuing ed- 

one of the participants indicated the federal trend by stating, ’’....There are 
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many hopeful si^ns fee improvement of (continuing education) programming, because 1 think 
the mood now Is '.efinltely to look for accomp 1 ishr.ient s and importance, to look at the 
objectives. What do you reaxly p^an to accomplish with X arricunt of staff and X amount of 
money and X amount of materials?'^ One need only lo look at grant guidelines, talk to 
foundation officials, or in North Caroline* to review the new budget request procedures 
to see the application of ti'ir approach and estimate its in.plications to planning and 
development . 

There is much to be optimistic about in this trend toward more rationalized decision- 
making. For one thing, it will place a needed spotlight on programs which have not 
had the Judgment and introspection to attempt to measure their achievements and identify 
their shortcomings. On the other band, there is good reason to be cautious and discrim- 
inating in what we buy from the new management science. The greatest danger is in being 
overwhelmed by the complexity and to escape either by embracing any method with numerical 
claims or by running at the first hint of scientific decision-making. Powerful analytica*. 
tools are being developed which can be valuable to continuing education, out it remains 
a world in which the saying caveat emptor applies as well as evet . Application of these 
techniques to education involves serious methodological and conceptual difficulties, ”once 
(as one author put it) you get beyond the level of easy majestic generalities.” While 
taking the side of objective, rational investigation, systems analysis involves judgment 
in every stage from problem definition through model building to estimation of parameters. 

Evaluation Objectives * 

Investment in evaluation should follow the same rules we should apply to any planning 
effort. The firwt question should be: ” What will this study accomplish and what dif- 
ference will it make to decision making ?” In my own experience there seems to be four 
possible objectives in evaluating continuing education programs; 1) to improve course 
design, 2) to improve teaching methodology, 3) to contribute to knowledge, and 4) to 
justify the organizational cr societal value of the effort. Some writers in the field of 
evaluation specify that objectives should be attainable and measurable. I would not 
force evaluators to undergo the same test, although I would urge that the objectives be 
clearly stated, meaningful, and as attainable and measurable as possible. Insistence on 
stating the attainable when you really want to go further is to reduce your true aim. 

Insistence on quantification when you are dealing with an Intangible is to settle for 
what may be an imprecise attribute of the real object in question. Although it is easy 
to understand why such mandates are included in the evaluation literature, they have two 
drawbacks: 1) they can tend to mask the real intentions of the program in question, and 

2) they encourage the use of implicit rather than explicit assumptions. I would prefer 
that the minister who wants to ’’save souls” keep that objective and assume that church 
membership is a reasonable estimate thereof, rather than that he restate his objective 
in terms of maximizing church attendance. In the former case he knows where his assump- 
tion is and can be sensitive to its influence on his decisions. Reference No, 6 gives 
some valuable insight on this point. 

Alternative Approaches 



Wha t are t he alternative approaches to accomplish the objectives ? There is a 
developing literature on this subject, much of which will likely be covered in the other 
position statements. In the Appendix I have specified five attributes which might be 
measured, depending on the objectives of evaluation: 1) exposure, 2) quality or present- 
ation, 3) change in knowledge, skill, or attitude, 4) behavioral change, and 5) end 
result change. It may be a moot point but I would like to reserve the term end Result to 
patient health status and to look at behavioral change as a necessary precondition. The 
reasons for this are twofold: 1) technically patient care is an activity, not an end 

result; 2) isolated focus on behavior does not shed much light on the priority importance 
in terms of the o/er-all health of the target population. Conceivably, extensive efforts 
mistht focus on a relatively trivial behavioral set, especially If given the mandate to 
( ^ ,.as priorities those activities which are more quantitatively measurable. 
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VJhat is the probable effectiveness and cost of each alternative ? This queotion is 
key to rational decisi^jii-making on the economics of evaluation. If one pursues it 
seriously, it becomes obvious that there is very little relevant data on the cost-effec- 
tiveness of evaluation. In fact, faith in the ultimate worth of evaluation is one of 
the primrry Justifications as it is in the field of education. Two authors who surveyed 
the evaluation of health programs in 1968 had this to say after reviewing 221 articles: 

"A thorough and exhaustive survey of the literature has led the authors to conclude that, 
first, evaluative research is ttill in its infancy; and, second, most publications on 
evaluation of health programs ejitoll the virtues of evaluative research but offer little 
operational guidance. The paucity of available research may be imputed to the lack of 
funds or lack of interest but it is more probably a reflection of the lack of valid and ^ 
reliable measuring instruments and the formulation of health goals in Utopian language." 

Certainly the literature scanned in preparation of this paper was relatively un- 
productive . 

Obvloujly, any meaningful judgment as to the effectiveness of continuing education 
evaluation efforts should be tied to the objectives of the study. Existing methodology 
can and does prc ' useful in improving course design and educational opprcaches. And, 
although much maligned, exposure studies are still extremely useful in justifying the 
apparent contribution to society even if the focus is on counting input rather than 
measuring output. Cost is a highly relevant factor in evaluation performed to meet the 
objectives of improving course design, improving teaching methodology, or justifying 
program contribution. In these cases the proportional expenditure for analysis should 
be viewed in the context of over-all program costs with some allowance tor serendipity. 
Cost is less relevant to evaluation done to meet the objective of contributing to general 
knowledge about educational approaches or measurement tcduiiques. The guiding principle in 
judging how much should be spent for evaluation is that marginal costs should approach 
but not exceed the expected marginal value of the information obtained. 

The Appendix to this paper was developed to answer the basic question of what kind 
of costs Vfc are talking about at various levels of evaluation, A one-week short course 
is outlined at an estimated cost of $42,000. Actual costs would vary with course design, 
content, and the participant group. It is interesting to note that participant time is 
by far the most costly item. The following table shows the estimated costs of five dif- 
ferent levels of evaluation (details are included in the Appendix) . 

Estimated Cost of Different Evaluation 



Approaches For A Hypotehtical One-Week Short Course 



Evaluation Approach 


Estimated 

Cost 


Fixed 

Course 

Cost 


Percent of 
Course 
Cost 


Participant Opinion Analysis 


$ 945 


$42,000 


n 


Pre-post Examination of Knowledge 


2,455 


42,000 


6% 


Mall Survey of Performance Change 
Survey and Observation of Behavior 


47,630 


42,000 


113% 


Change 


155,850 


42,000 


371% 


End-Result Studies of Patient Health 


212,350 


42,000 


506% 



There Is a great deal of approximation involved in such estimates, but not much 
more or less than that involved In developing the actual budget of a research proposal. 
The participant opinion analysis and pre-po^ examination are within an acceptable range 
In proportion with course costs. The studies of behavioral change, or patient health 
status, greatly exceed course costs and could be justified only as they relate to the 
objective of contrfbuting to general knowledge. 

o 




40 



Selecting An Evaluation St i ;ategy 



2 



What evaluation strategy, or mix of strategies, will m aximize gain In term s o f the 
value of inforroation received while minlTnizing cost ? Rarely could an evaluation strate iy 
be selected on a purely quantitative basis. But the point of view of attempting to max- 
imize gain while minimlziiig cost is a useful ideal. There is a danger in this approach 
of looking only at short-range benefits where the real pay-offs might occur in the long- 
range future. The final choice should involve a close look at the objectives of evalu- 
ation, the likelihood of obtaining useful data, and the estimated costs. There is no 
general rule as to what cost level is appropriate as compared with over-all program costs- 
Nick Parlette suggests the general idea of setting aside a 5 to 10 percent override for 
evaluation, some of which might be accumulated for several Intensive research programs 
aimed at attempting to build evaluation skills and perform end-result studies. The 
general idea of this approach aeents tempting, although the proportion allowed for evalu- 
ation may be small given the lack of reported research at this time. However, it would 
be misleading to suggest that such a percentage be spent for evaluation of each course. 

It would be fruitless to perform studies which would yield course improvement data for 
nonrepeti tive ventures. It would be equally futile to extensively evaluate a given pro- 
gram when it WeS apparent in advance that the data would not be utilized. On the other 
hand, certain courses might lend themselves to rigorous evaluation aimed toward building 
educational theoriec. In these the evaluation expenditure could justifiably exceed course 
costs many times. 



We are currently entering a stage in which highly rationalized, quantitative deci- 
sion-making will likely guide the resource allocations of most federal, state, and local 
governments, This can be expected to have a definite effect on continuing education pro- 
gram development which, given our lack of past evaluative efforts, may be of some real 
benefit. It is important to view this trend with a mixture of optimism and caution, for 
it could lead to significenit improvement in the quality of decision-making, or it could 
lead toward a rather undesirable state of mechanized mediocrity. Perhaps one of the 
more noted commentators on the public issues involved is C. E, Lindblom whose article, 

’’The Science of Muddling Through,'^ published in 1959 is still a classic analysis of de- 
cision-making in a pluralistic society. 

The primary difficulty of analyzing the economics of evaluation xn the health field 
occurs In assessing effectiveness values rather than in estimating costs. One author 
hints at the difficulty in an interestl; g attempt to correlate increase in health expendi- 
ture with longevity and concludes that; ^'It is probable that we could either halve oj^ 
double the money now spent on health without significantly affecting our longevity." 

His results technically reflect the fact that longevity Is no longer a sensitive index 
In this country (neither, for that matter, are most morbidity or mortality data). The 
lack of a market-place, the small role of consumer choice, the noncompetitive orientation, 
and the prevalence of intangible, noneconomic values which resist quantif icatij^ all lead 
to extreme difficulties in analyzing the benefits of improved health services. In 
terms of continuing education, there is even great difj^iculty in deciding whether costs 
should be treated as assets (investments) or expenses. 

The uncertainties involved in measuring benefits in the health field make it im- 
possible to arrive at a completely objective judgment as to appropriate evaluation costs. 
Given this situation, it is convenient to borrow a closing remark from a session on the 
Programming, Planning, and Budgeting System and quote Jean Paul Sartre: ’’These are 

difficult times; one does what one can." 



Summary 
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Appendix 

Cost Es ti mates cf Alternative Evaluation Techniques 



Introduction : 



This appendix has been developed in order to provide a basis for cost comparison 
among alternative evaluation techniques. Its purpose is to provide a response to the 
question of what kind of costs are involved at specified levels of evaluation. Such 
costs are not fixed amounts subject to precise definition but are reasoned guesses of 
resources required for specified research approaches. 



Individual cost estimates for five research approaches are generated, assuming a 
hypothetical short course of one week’s duration. The rerulting data are useful to a 
discussion of economic allocation Issues and priorities. 1 have attempted to be conserva- 
tive in the estimates to avoid any inclination to exaggerate such costs. Anyone desiring 
to use these data for budget estimate purposes is alerted to examine the figures carefully. 

Evaluation Alternatives ; 

There are a number of alternatives for evaluating a short course, depi;ndlng on the 
purpose of the evaluation and the related attributes to be neasured . 



The reasons we might consider for evaluating a short course Include: 



1. De /elopmental Evaluation — To improve course design and presentation. 

2. Normative Evaluation -- To compare different instructional approaches In order to 
choose the most cos t/ef feet Ive educational strategies. 



3. Inductive Evaluation — To contribute to knowledge or build methodology related to 
continuing education. 
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O olute Evaluation — To justify the societal 
compared with other resource allocations. 



contribution of 



continuing educatlc.i 
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The attributes which could be measured and illustrative evaluative approaches include: 

1 . Ex posure (Outreach ) : 

a. Differential stuly of participant distribution and characteristics, 

b. Motivation and interest studies, 

2. Qual i ^ of Presentatit n (Input) : 

a. Faculty judgment. 

b. Participant opinion ant^lysis, 

3 . Assimilation (Knowledge, Skill, and Attitude Change) ; 

a. Pre-post examination, 

b. Critical incident studies. 

4. Job Performance (Behavioral Change) : 

a. Mailed questionnaire. 

b. Field survey. 

c. Professional survey anc observation, 

5. Societal Result (Output ) : 

I 

a. Judgmental estimate and analysis, 

b. Direct observation and measurement. 

Most reported evaluation efforts have been concerned with differential exposure 
analy.3ls> studies of the quality of presentation, and examination of participant 
assimilation. Lately, the developing sophistication of educational theory and research 
and the increased weight of the management science viewpoint in governiaert decision - 
making have served to change this emphasis. As pressure increases for greater attention 
to evaluation, the objectives and comparative cost must be considered in terms of over- 
all program goals. 

The following estimates have been developed given the lack of empirical data on 
this topic. 

Course Costs : 

Suppose we were concerned with choosing an appropriate evaluation approach for a 
one-week (five-day) short course for fifty physicians. The direct and indirect course 
cost might be estimated as follows: 



Instructional Costs (Assumed to be $40 per day per student) $10,000 
Participant Expenses ($20 per day per student) 5,000 
Participant Travel (Assume Average Distance 400 miles) 2,000 
Replacement Salary or Income Loss (Assume Average Annual 

Income of $24,000) 25 ,000 
Total Course Cost $42,000 



The actual cost of a given course would of course vary widely depending on duration, 
content, type of participants, location, and material involved. The above estimate should 
be reasonably representative of the cost of the general short course in the medical field. 

Evaluation Costs ; 

Given the four purposes of evaluation (course improvement, knowledge change, be- 
havior change I and end-res ilt change) we might select the following evaluation approaches 
^ cified cost levels. . ^ 

cKje 
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1. Participant Opinion Analysis (Quality of Presentation) ; Participants are asked to 
ricord their Interest in each presentation, estimate its value to their performance, 
and make general suggestions as to course design and content. Ihe results will have 
relevance to redesign of the course if repeated and to general course planning. A 
number of react ionnaires designed to collect this type data are available, thus 
development is largely a matter of adaptation. 



Professional Design (1 executive day) $ 80 
Participant Cost (3 man days total) 330 
Analysis (5 math aide days) 175 
Report Preparation (2 executive days) 160 
Support Personnel, Materials, Distribution 200 

$945 



2. Pre-Post Knowledge Examination ; Participants would be asked to complete an objective 
content examination at the beginning and upon completion of the course. It is assumed 
that construction of the test would involve two fifty-question instruments of equal 
difficulty. The examination would be pretested and refined including verification of 
results prior to use. It would be administered to a small cciitrol group in order to 
see if the examination itself stimulated learning. The results of any given pre-post 
knowledge examination are difficult to interpret except in terms of estimated rela- 
tionship to given objectives. However, the results can have high normative value 
when data from several courses or educational approaches are used for comparative 
purposes. 



Professional Design (5 M.D. Consultant days) $ 550 
Test and Measurement Design (2 Psychologist days) 140 
Pretesting, Refinement, and Development (includes control 

and refinement) 500 
Administration and Analysis (5 Math Aide days) 175 
Participant Time (6 days) 660 
Statistical Analysis (3 Statistician days) 210 
Report Preparation (2 M.D. Consultant days) 220 
Support Personnel and Report Preparation 300 



$2,755 

3. Mailed Questionnaire Study of Performance Chan ge: Six months after attending the 

course, participants are asked to: 1) give their opinion of performance change; and 

2) respond to diagnostic and treatment alternatives related to critical patient care 
incidents. The latter would Include situations covered in course content and situa- 
tions omitted from course content. Each respondent would be matched with a non- 
participant for age and type of practice for control purposes. Questionnaires would 
be sent to the fifty participants and an equal number In the control group. Adequate 
and inadequate response patterns would be developed by an expert panel of medical 
consultants and the test Instrument thoroughly pretested prior to its administration. 

It is assumed that agreement to participate In the post-course study would have been 
a condition of enrollment. Given sufficiently discrete course content, there might 
be a possibility of obtaining results which would be useful in estimating the degree 
of success in achieving stated behavioral objectives. 

Professional Design (30 M.D. Consultant days) $ 3,300 

Experimental Design Team (Statistician, 

Psychologist, Sociologist, Operations 

Analyst - 60 days total) 4,200 

Project Director (168 M.D. Consultant days) 18,480 

Math Aide (168 days) 5,880 

* The personnel costs are derived from the following assumed salary levels based on data 
reported in '’National Science Foundation," American Science Manpower, 1968 . Washington 
D.C.: U,S. Government Printing Office, 1969 (Reference 1). Specialist Medical Consultant - 

$110 per day; Executive Administrator - $80 per day; Statistician - ^70 per day; Psycholo- 
^ $70 per day; Sociologist - $65 per day; Economist - $65 per day; Operations Analyst - 

r day; Interviewer - $3$ per day; Math Aide - $35 per day; Secretary - $22 per day. 




Statistical Analyst (60 man days) 
Participant Cost (12 days) 

Computer Programming 
Computer Operation 
Secretarial Support (225 man days) 
Materials and Distribution 



4,200 

1,320 

2,000 



4,950 

^500 



800 



$47,630 



4. Professional Survey of Patient Records and Observation : In order to obtain baseline 

data, prospective participants would be interviewed and patient records studied upon 
enrollment in the course and again one year after attendance. Identical interviews 
spaced one year apart would be conducted with a smaller control group (say ten phy- 
sicians) , egch of whom would be matched with a reference individual in the partici- 
pant group. Assuming about 3,500 patient care incidents per participant per year, 
a total of aDOT^t 210,000 record entries could be expected per year including the 
control group- Selecting even one percent of these for study of special clinical 
entities would require scanning of the entire number of entries and detailed review 
of 2,100 cases for each of two years (given the design of comparing diagnosis and 
treatment patterns over the year prior to the course with those in the following 
years). Information gained from review of records would be supplemented by interview 
and direct observation in practice situations. This type of study would have to cor- 
rect for bias among the physician group willing to participate and the nonpart icipant 
in order to generalize about over-all course benefits. The nature of course content 
would have to lend itself to measurement such as involving discrete and relatively 
new treatment patterns or diagnostic tedmlqucs. The atypical nature of such content 
would also need to be considered in generalizing about study results. The information 
gained by such a study would provide direct feedback on achievement of behavioral ob- 
jectives specified in course design. It would be particularly valuable in the in- 
ductive sense of contributing to insight on evaluative research merh^dolo'^y or general 
knowledge about effectiveness of continuing education. 



5. Direct Measurement of Societal Result ; For purposes of cost estinaciLiu i'- is assumed 
that an end-result study of patient health status would be preceded by amlysis of 
participant behavioral change and that thereafter it would have many of the character- 
istics of the more familiar drug study. Altered physician behavior would be handled 
as a treatment pattern (e.g., experimental drug) and evaluated in mud :be same way. 
Such an approach would only be practical for a specific clinical entity involving 
discrete behavioral patterns and measurable change in patient health status (as related 
to the clinical entity in question) . A 10 percent sample would be drawn from the 

* The estimates allow the record analysis scanning time of 100 record‘d n*r hour and 
review time of one-half hour each for 2,100 records. A physician v id be re- 
^ “uired to conduct interviews and observe practice patterns with the a-=:sumptlon 
,e of two interview-observation events could take place aach day. 



Experimental Design (90 Senior Analyst days) 
Professional Criteria (40 M.D. Consultant days) 
Project Director (2 years) 

Educatiorial Psychologist (18 months) 

Record Analysis (200 man days)* 
Observer-Interviewer (90 M.D, days)* 
Statistician (100 days) 

Participant Cost (1 day each, 50 days) 
Secretaries (3 years) 

Math Aides (2 years) 

Computer Programming 
Computer Operation 
Transportation 
Materials and Distribution 



$ 7,200 



4,40r 
40,000 
27 ,000 

7.000 
9 ,900 
7 ,(X.J 
3,500 

14,850 

16,0il0 

8.000 
1 ,000 

5.000 

3.000 



$155,85^ 




added . 
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patient records identifier in the study of behavioidl rhange* including a matched 
control group, for a total of 420 persons. Patients in the sample would be inter- 
viewed by a physician who was not aware of whether the subject was a member of the 
control group or study group and an estimate made of his health state, as related 
to the given condition. This assumes that interview and observation would be suf- 
ficient in this case. The potential value of the information from a thorough study 
of this sort would be extremely high, assuming that a course could be found which 
net the necessary specifications. Conceptually, such a study could contribute to an 
understanding of the nature of the association between educational activity and end 
result, could measure the validity of lesser evaluation techniques, and could provide 
some basis for computation of educational benefits. In reality it night be difficult 
to find a situation in which the stu:’y could be conducted. 



Study of Behavioral Change $150,350 
Additional Design Efforts (30 days) 2,400 
Added Project Staff Time (1/.? year) 22,000 
Participant Cost (1 day each, 50 days) 5,500 
Pa.ient Interviewers (210 M.D. days) 23,100 
Added Analytical Time 5,000 
Transportation 4 , 000 

$212^350 



Cost Comparisons : 

The cost implications of the alternative evaluation approaches appear when compared 
with over-all course costs and possible utility factors. If the figures are anywhere 
near correct, evaluation costs could be expected to run from about 2 percent to 500 per- 
cent of the total course costs. It is obvious that the behavioral studies and end-result 
studies can only be justified in terms of the potential contribution to knowledge about 
educational design or development of evaluation methodology. The costs of doing these 
studies for course refinement, course justification, or comparison of alternative ap- 
proaches is far too excessive. 



Evaluation 

Approach 


Total 
Estimated 
Cost ($) 


F ixed 
Course 
Cost 


Percent of 
Course 
Cos t 


Potential 

Utility 


Opinion Analysis 


945 


42,000 


n 


Developmental 


Pre-Post Test 


2,455 


42,000 




Developmental, Normative 


Mailing Survey 
Performance 


4^630 


42,000 


113% 


1 nduc t ive 


Behavior Observation 


155,850 


42,000 


371% 


Inductive, Absolute 


End-Result Studies 


212,350 


42,000 


506% 


Inductive, Absolute 



It is important to note that course costs have been arbitrarily held constant in 
this example. In reality, these costs could vary widely, while it is reasonable to expect 
the cost of the described evaluation efforts to remain relatively stable. There would be 
much less variation in the cost of the behavioral observation approach for a one-day 
course versus a three-week course than there would in the direct course costs. 



The actual decision as to the level of evaluation most appropriate to a given 
course would involve a number of consl- irationv: the purpose of the evaluation, the 

probability of being able to measure desired output, the time allowed, the receptivity 
of the participant, the availability of measurement instruments, and the over-all cost 
as compared with course cost and expected value of the information. This last factor 
is undoubtedly the most Important and, like many other aspects of education, does not 
readily lend Itself to quantitative treatment. To the extent that evaluation has the 
purpose of describing the contribution of continuing education to society, each general 
approach has specific strengths and limitations. Documentation of participant exposure 
is inexpensive but can be related to a change in societal health state only by an un- 
certain chain of conjecture and assumptions. Estimation of behavioral change through 
field observation of participants would be extremely expensive and subject to numerous 
'"‘““^■‘•^ement errors, but it would greatly reduce the assumptions involved. 
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In the final analysis on*^ cannot escape a basic dilemma common to the human service 
persuasions in our mechanized culture. On one hand, society is asked to finance ex~ 
tensive educational efforts with very little documentation as to their value. On the 
other, if it tries to correct this by research and analysis, it ultimately must face 
underwriting extensive studies with very little documentation as to the value of the in- 
formation obtained. The obvious recourse in the face of these uncertainties is for 
society to give greater attention to the tangible and more measurable options. The only 
practical response is to attempt to develop better estimates of program effectiveness, 
to argue the intrinsic value of the more qualitative and less measurable aspects of human 
endeavor, and to bring the best minds to bear on questioning the apparent but deceptive 
value of the more tangible and visible alternatives. If there is any success, high 
quality human service might at some time attain the political and social appeal now 
reserved for our massive highway systems, space programs, defense capabilities, and im- 
pressive building program. 
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MEASUREMENT PROBLEMS 
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Hugh M. Petersen, Ph.D. 



The major problem associated with the evaluation of continuing medical education is 
that of establishing the reliability and validity of measurements. Most of us know 
something about the statistical definiticns of these two terms. However, I'm sure there 
are those who have not realized that we have initially a problem in semantics. That is, 
in a real sense we have a problem of knowing the true meaning of our measu: ements as well 
as that represented by their statistical reliability and validity. 

This problem may be exemplified by considering the reliable and valid measurement of 
"learning.^' Most of us would be willing to hypothesize at a theoretical level that 
"learning" by practicing physicians, through continuing educational opportunities, should 
improve patient cart We might go beyond the hypothesis in the direction of tacit as- 
sumption. In either case, we are operating and communicating at the factual-meaning, 
or somewhat abstract, level. 

At this level the meaning varies depending mostly upon the level of abstraction 
employed. Many of us would agree as to some of the activities which serve as illus- 
trations of learning; e.g., acquiring a medical vocabulary, memorizing the parts of human 
anatomy, and acquiring surgical skills. There are other activities which are more subtle 
examples of learning, such as the acquisition of medical prejudices and preferences, as 
well as other social attitudes and ideals. Perhaps we should be satisfied to define 
learning as "improvement with practice," or "profiting from experience." Vet, we know 
quite well that some learning is certainly not improvement, and other learning is not 
desirable in its consequences. Maybe a better approach is to define learning as a "change 
in performance resulting from some specified educational experience," None of these 
definitions is really acceptable because the level of abstraction employed introduces too 
much ambiguity. We really don't know what these definitions mean until we reduce the 
ambiguity inherent in the various terms utilized in their construction. 

At the factual-meaning level our communications are fraught with ambiguities; i.e,, 
we don’t really know what is meant by "learning," by "continuing educational opportun- 
ities," by "Improve," or by "patient care." If each of us were asked to define these 
terms, much variation would result from one person to the next. This fact of ambiguity 
directs us to look for a more concrete, "point-at-able" level of communication which 
insures considerable invariance in meaning. 

Educators and students often use concepts that are formulated at rather high levels 
of abstraction. These are quite different from those utilized by the empirically minded 
educational researcher who operates at the "point-at-ab le" level. In attempting to over- 
come the differences between these two levels, it is suggested that we refer to the latter 
kinds of variables as indicators of the former concepts. In so doing we adopt the posi- 
tion of operat ionism . 

The philosophy of operationism and the idea of defining the abstract In operational 
terms is commonly associated vith the physicist, Percy Bridgman, writing In his work, 

The Logic of Modern Physics . Arthur Eddington, the astrophysicist and philosopher of 
science, gives us some insight into the need for operational definitions by means of the 

following problem. "An elephant slides down a grassy hillside " The mental picture 

Is probably stimulating, but that is irrelevant. Reading on, we find that the mass of 
the elephant is two tons. At this point we may ask what is, or are, two tons,' Do the 
two tons belong to the elephant? If ve think of two tons as property of the elephant — 
confusion of meaning results. Historically, physics textbooks have defined mass as a 
"quantity of matter." However, it was only opinion that such a quantity of matter 
might be represented by some accepted measurement techniqu'. Present-day physics 

O s would not bother with opinion. They know that mass Is inferred from "point- 
s’’ such as meter or pointer readings. 
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All other variables inherent in the elephant's hillside slide, e.g., coefficient 
of friction, hill slope, and descent time, may be defined by pointer readings. The 
lesson to be learned is that, even if we have definite ideas and beliefs regarding "real” 
objects existing in the ex:ernal world, these mental representations are useless except 
as handled by the science of operationism We cannot handle the measurement problem until 
the concepts or constructs are replaced by specific operations, 

George Lundberg, who championed the operat ionalls t position in sociology, strongly 
defended the premise that all variables are measurable and that one should not be con- 
cerned with hypothetical entities. He further claims that it is erroneous to assume that 
measurement is not a way of defining things; i-e , it is erroneous to assume that measure- 
ment is a process which can be carried out only after the "thing" to be measured has been 
def ined , 



Eddington stresses that variables should be defined according to the way they are 
recognized, and Bridgman points out chat the proper definition of a concept is not in 
terms of properties possessed, but In terms of actual operations. Therein lies the es^- 
sence of operationism. It means that, if variables are defined in terns of properties, 
there is no possible way of testing directly any hypotheses in which the variables appear. 
Tests and experiments must be performed in terms of operations; it is the "point-at-ables , " 
i.e., the pointer readings, that are related. 

The essence of operationism consists in reducing a situation to elements 
we are so familiar and unambiguous that we accept them as a matter of course, 
curiosity rests and overconfusion is reduced. 

The use of operational definitions in describing the variables associated with 
learning insures reliability in the sense that others may duplicate our system of mea;5ure- 
ment . Thus, in measuring learning, we must devise some meter, representing an objective 
scale, which we monitor before and after an educational experience In order to determine 
whether learning has occurred, we would compare pointer readings as represented by pre- 
and post-test scores. If a change in pointer readings results, we subject the change to 
a statistical test in order to assess the probability that chance was the major determiner 
of change. If the probability of chance being the source of change is small, we conclude 
that the change is due to learntng. 

Summarizing to this point we see that .cncepts and constructs are useless in edu- 
cational measuremert unless they are operationally defined. The act of operationally 
defining insures tha realiability and replicability of our measures 

The positive relationship between operationism and measurement reliability does not 
necessarily hold for measurement validity. For example, we may operationally define 
learning as a statistically significant positive change in test scores from pre- to post- 
test experiences. If our test items have been well designed, chances are they ate 
statistically reliable. Since the test has ^een unambiguously .specified, logical reli- 
ability as Well as replicability has been assured. The validit}' of our measure of learn- 
ing is something else. 



with which 
so that our 



More and more we are beginning to accept the fact that there is no real validity in 
academic measures of learning in medicine, except as may be demonstrated within the con- 
text of patient care. The only valid system of measuring learning involves the measurement 
of the predisposition to apply course content in job performance. Ke must be able to 
monitor physician-patient interaction and the results of such interaction before we 
know anything of substance regarding the effects of medical education experiences. 



Let us consider the case of postgraduate courses in hematology* The University of 
Kansas has prepared a course of study in^heroato logy which is well known and respected 
by the many physicians who have used it Much work has gone into the preparation of 
the course, and yet that work cannot be properly evaluated by restricting our evalu- 
ation to the monitoring of pre- and posttest measures of the usual paper and pencil 
type. This type of evaluation does not do Justice to whe course or the physician- 
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It is suggested that physician job-performance measures be designed for each contin- 
uing education experience. These measures would indicate the extent to which the 
physician applies course material he has studied in any educational program. 

As a specific example of the measurement of physician application of principles and 
techniques learned in a course in hematology, the following evaluation instrument is pre- 
sented . 



Hematology Course Evaluation 
(Parameters to be recorded) 

A. General Information 

1. Attending physician 

A. G.P. 

b. Int . 

c. Other Specify 

2. Case number 

3 . Race 

Sex: Male Female 



5. 


Age: <10 


10-20 


21-^0 


^1-60 


>60 


6 . 


Consultant use: 


Yes 


■ No 






7. 


Admission date; 


Day 


Month 


Year 




8. 


Discharge date: 


Day 


Month 


Year 





9. Number of hospital days; 

10, Death: Yes No 

11. Admitting Dx 



12. Discharge Dx 




1)0 



B, History: 



1, Complete? Yes No^ 



Bleeding recorded? Yes 


No 


Not mentioned 


a. If 


yes: 






1) 


Source recorded? 


Yes 


No 


2) 


Amount recorded? 


Yes^ 


_ No 


3) 


Color recorded? 


Yes“ 


No 


b. If 


patient is female: 






1) 


Menstrual history 


recorded? Yes No 


Hemorrhage into skin recorded? 


Yes No 


a. If 


yes : 






1) 


Extent recorded? 


Yes 


No 


2) 


Duration recorded? 


' Yes 


No 



4. Dietary history recorded? Yes No_ 

a. If yes: 

1) Type recorded? Yes N o 

2) Amount recorded? Yes___^ No 

5, History of drugs recorded? Yes No 

a. If yes: 

1) Type recorded? Yes No 

2) Amount recorded? Yes No 



6, Jaundice recorded? Yes No 

7, Family history of anemia recorded? Yes No_^ 

8, Family history of bleeding recorded? Yes No 

9, History of weight change noted? Yes No 

C. Physical Examination: 

1. Presence or absence of pallor noted? Yes No 

2. Presence or absence of skin hemorrhages noted? Yes_ 

3. Presence or absence of active bleeding present? Yes 

Blood pressure recorded? Yes No 

5. Pulse recorded? Yes No 

6» Presence or absence of "spider” angiomata? Yes 

7, Tongue described? Yes No 

8, Liver size noted? Yes No 

9, Spleen size noted? Yes No 

10, Lymph nodes size noted? Yes No 

11, Edema (swelling) of legs and feet noted? Yea No 

Neurological examination done? Yes No 




D. Hematological Consultant: 



^0 






1. Was one used? Yes No 

2. Consultant’s diagnosis? 

a. Does It conform to that of admitting physician? Yes No 



E. Laboratory Orders: 



1. CBCs (or fractions thereof). How many ordered? 



2 . 



3. 



4 . 



5 . 



6 . 



7. 



8 . 



9. 



10 . 



11 . 



12 . 



1 ^. 



15. 



16. 



17. 



18. 



19. 



20 . 



21 . 




Was fecal occult blood test ordered? Yes No 

Was there an order for fecal examination for parasites? Yes No 

Was total serum iron ordered? Yes No 

Was iron-binding capacity ordered? Yes No 

Was Coombs’ test ordered? Yes No 

Was haptoglobin ordered? Yes No 

Was reticulocyte count ordered? Yes No If yes, how many?_ 

Was hemoglobin electrophoresis ordered? Yes No 

Was serum protein electrophoresis ordered? Yes No 



Was 


gastric analysis 


ordered ? 


Yes 


No 


Was 


serum folic acid 


ordered? 


Yes 


No 


Was 


serum B-12 assay 


ordered? 


Yes 


No 



Was Schilling test ordered? Yes No 



Was red-cell osmotic fragility test ordered? Yes No 

Was glucose-6 phosphate dehydrogenase activity ordered? Yes No 

Was pyruvic kinase activity ordered? Yea No 

Was bone marrow study ordered? Yes No 

Was liver biopsy done? Yes No 

Was lymph node biopsy done? Yes No 

Was Initial screening procedure for hemorrhagic diathesis? 



a. PT: Yes No^ 

b. PIT: Yes No 

c. Bleeding time: Yes No 

d. Clotting time: Yes No 

e. Platelet count? Yes No_ 

f. Thrombin time? Yes No 

g. Clotting time? Yes No 

h. Clot retraction? Yer No_ 



1. Test for flbrlnolysi* 3? Yea No 

j, Prothrombin consumption time ordered? Yes_^ No 

k, Was fibrinogen level ordered? s No 

l, Was chromosome culture ordered? Yes No 

tn. Was leukocytes alkaline phosphatase ordered? Yes No 
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This hematology evaluation form probably needs some refining, but it represents a 
concrete attempt at operationally defining the learning which may have taken place as 
the physician has studied. The comparison of pre- and postcourse physician behavior 
would yield some unambiguous pointer readings, representing validity far beyond that 
inherent in paper and pencil tests. 
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SECOND PLENARY SESSION 



CHAIRMAN RISING: It is the plan, subject to your pleasure, to devote thxS morning's 
session to thirty-minute presentations by each of the discussion groups of yesterday. 
During that time I am going to turn the lectern over to the Group Consultants, with the 
exception of Steve Abrahamson who had to leave, in which case Bill Nelson has kindly 
consented to be the substitute Group Consultant. The Group Consultant may use his time 
in any way he sees fit. He may give a crystallization of his group's discussion of 
yesterday, he may ask somebody else to give it, or he may call on sundry members of the 
group, even all the group, for their contribution, 

This afternoon members of other groups who want to enter arguments, other ideas, 
modifications, etc. will be free to get into the discussion. So, unless I hear a 
majority wishing to change that, we will reserve the afternoon for general discussion 
by everybody here. We hope to have all of the different points of view represented by 
the persons of varying backgrounds and Interests reflected both this morning and this 
af ternoon. 

Group One will give the first report. 

DR. NELSON: It was a challenge to put our group's discussion into some sort of 

order. We covered a lot of territory. What I would like to do is put it under several 
headings and give what I think was said in each of several different topic areas. 

The topic of the day was "Obstacles to Continuing Education" and our particular 
group was supposed to be concerned about "Inappropriate or Unrealistic Goals." As you 
recall, Steve presented some of those to us yesterday morning. 

The net result was probably a discussion, if I may paraphrase it, of various 
problems in pursuing continuing education programs, and perhaps "problems" is the same 
as "obstacles" but I am not sure it is exactly the same. 

The coniinents fell into five main areas: 

1. We talked about motivation, or lack thereof, to pursue continuing education. 

2. We talked about the problem of wants versus needs — that old saw that we have 
been down the road a good many times on — the wants versus the needs for contln irp 
education, and both the recognized and the unrecognized needs. 

3. We talked briefly about the pressures that are developing toward evaluating 
wants or needs, not at length but at least we recognized there were some pressures in 
this direction, and I think it is very Important. 

4. V’e talked about techniques for evaluating needs. I want to point out we did 
not discuss evaluating the educational programs themselves. 

5. Very briefly there were some comments on techniques of delivering continuing 
education, of meeting the needs. It was brief but it focused on some interesting things 
in the national picture. 

So, if I may, I would like to report on these five categories. I an sure members 
of the group will have comments to make In amplification of some of the points that we 
discussed . 
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Motivation was the area in which we started ^ Someone asked the original question, 
"Isn't motivation the main problem?" Then they or someone else went on to indicate that 
the educational opportunities are there for continuing education for physicians, in particu- 
lar, but asked whether it isn't a question of motivation whether physicians take advan- 
tage of these opportunities. Indeed, don't we see, really, that physicians may pursue 
continuing education in medicine, but they also may pursue golf or they may pursue stocks 
and bonds, as somebody put it, or many other things. Isn't motivation, then, the question? 

This led to a discussion of the problems of priorities on the part of physicians 
for their time. They are concerned with these other things, but they also are concerned 
with delivering good quality care too. There is a variation in this, but the physician 
is faced with a balance in terms of the many demands on his time, the "busyness" of his 
practice, the need to have relaxation versus the question of pursuing it in terms of 
continuing education. 

Steve made a very strong point that he felt most physicians do want to practice 
good medicine. I would like to emphasize that I felt strongly this was true. I believe 
most physicians do want to practice good medicine if it is possible, and they are moti- 
vated to want to improve. However, they are faced with this question of priorities and 
to what degree they want to put the effort into the continuing education efforts versus 
perhaps other things. 

Then Steve vent on to indicate that part of the problem he felt was inappropriate 
objectives In education and in the things that we offer that perhaps kill the interest 
{motivation) of physicians to be involved, A little bit later we asked questions about 
this, and I think what he really was saying is that many of the programs that are of- 
fered really do not meet the needs of the physician as the physician sees his own need. 

Someone brought up a list of interfering factors with regard to motivation. I 
will not recite them all, I am sure m^st of us know them, but they are the kinds of 
things that were in the Vollan Report of many years ago and the California study, and 
various other studies. They concerned the problems cf the "busyness" of his practice, 
his inability to get away, the finances, the timing, the availability of programs for 
what h£ was interested in, etc. There are many cf these factors. 

Another factor was the learning process. The study process itself is basically 
toil, it is work, it is effort, and to be involved in continuing education is to face 
up to doing something that is not necessarily a whole lot of fun. This raised the 
question that perhaps or^e of the ways to overcome some of this difficulty is to be sure 
that we think in terms of programs that are attractive. 

Finally, under motivation, one of the members noted that medical students under 
certain situations tend to shun taking advantage of the learning-study opportunities 
which they have but which are not required in their course efforts, and this led to the 
suggestion that perhaps in some ways we are tending to breed a group of physicians today 
and in the future who will be even less motivated than physicians in the past. 

Next we discussed wants and needs. In this category it was noted that physicians 
vary widely in their practice activities This is stating the obvious, but it is impor- 
tant and there were a lot of different factors brought in. "General Practitioner" is 
a very broad term and there are many kinds — the general practitioner versus the in- 
ternist, rural versus urban -- I don’t think it was put in those words but 1 think that 
is what it meant , 
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One very interesting point that cair.e up was that there are differences in what phy- 
sicians need to know in rural areas versus urban areas. This came up, incidentally, in 
connection with medical audit that we will comment on in a few minutes. Maybe there are 
differences here, One of the spokesmen who spoke effectively from the standpoint of 
rural practice said, ’'I do not believe that is significantly true. Today, with communi- 
cations and exchange of opportunities, with the mail for steroid determinations, etc., 
there isn’t much difference between rural practice and urban practice in terms of what 
we need to know," 

The second "wants and needs" area of discussion was needs that we hot; tend to ob- 
serve in analysis of hospital records. Indeed, are we not missing the boat because 
office records are very different, and yet ninety percent of all practice is office 
practice. I think it might be pointed cut that most medical groups are concerned with 
office practice. The American Society of Internal Medicine, as many of you knew, is in 
the process of making quite a significant study in Syracuse. 

Wants and needs are different, and Bob Keth made a couple of brief comments on this. 
I will not try to paraphrase what he said except that he pointed up the difficulties 
that may be involved in replying on the wants and the popularity expressions of individ- 
uals with regard to some of Che course programs in which they have been involved. 

Steve Abrahamson indicated that our thoughts regarding the physician’s wants and/or 
needs may be quite different from his , and this may lead to inappropriate objectives 
in our programming. 

There was a brief discussion regarding the role of basic sciences in terms of v?ants 
and needs. I will leave that for others to talk about. There was somewhat of a dif- 
ference of opinion about the necessity of being unduly involved in basic science. 

It was pointed out In regard to wants and needs that just the plain ordinary routine 
history and physical, and many other routine things need to be given more consideration 
than they sometimes get. In this regard certain individuals recalled the role-playing 
approach that the Board of Orthopedics is taking in terms of evaluating the capacity to 
do history and physical effectively. 

In addition to these wants and needs there were tw’o or three that were very inter- 
esting because they were a little bit off-beat and maybe we do not always think of them. 

In terms of this total list, somebody said, "You know, we have been discussing the 

medical tangibles, pneumonia, the capacity to handle the acute coronary, whatever you 

will. What about insights into the emotional areas? Not just psychiatry but emotion in 
relation to illness, in relation to life and death, in relation to conception, to family 
business, etc., the kind of things that we do not really see even in the records. These 
are the things that go on between a physician and his patient, or the patient’s rela- 
tives, Do we not need something in evaluating procedure here? 

It was mentioned just in passing that there are a number of courses for practicing 
physicians in relation to emotions, not necessarily psychiatric, and they are often 
well attended. One of the important ones that I know is Dr, Kauffman’s at Mount Sinai. 

We have them in Albany and I suspect many of you do too. In that regard it was asked, 
"How do you visualize a medical audit In this area?" 

Someone said that often the biggest gap is not in clinical knowledge but in office 
management. Physicians are being involved in a new role as executives, as one of Che 
individuals put it, in the health care delivery system. How do they answer the phone? 

How do they use their office personnel? How do they get more efficiency out of every- 
thing that they do? Well, it is off-beat, but at least I leave it with you as another 
area that we need to be a little concerned. 
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We vent on to the question about the wants and needs of the physician with regard 
to how they use office personnel to help them in educating their patients » This brought 
up the question of the use of ancillary aid3 as well as audiovisuals. As many of you 
know, in pediatrics, obstetrics and gynecology, in maternity care, etc., these are being 
very attractively and significantly used by some individuals, but there is a vast lack 
of knowledge on the part of the practicing physician, probably on the whole, in terns of 
this area. 

Another one was that many physicians do not know what is available in the variety 
of allied health personnel services. We all recognize there are a lot of special kinds 
of services that physicians often do not know are available to them. 

Pressure toward evaluating needs was the next topic. There were comments on the 
fact that there are real pressures building up. These are the third-party payors. 

(unions. Blue Cross, as well as other public altruistic groups) that are simply concerned 
with quality of care quite Independently of the finances, etc. 

This area led to comments upon the possible distinction beeween recertification and 
relicensure. There I 3 an important difference* Recertification labels the individual 
as a person practicing a certain kind of quality of practice. It is not an essential 
credential for practice. It is not a legal instrument. It is one of quoting the quality. 

It was suggested that maybe recertification with challenge examinations is appro- 
priate. 

In licensure, because it is a legal matter — a legal capacity to earn a living — 
there is pressure in a different direction, not for challenge examinations to relicense 
but for requirements that individuals be involved in continuing education experience. 

If they at least participate they can be relicensed. 

Next is the matter gf techniques for evaluating needs. Bernard Dryer’s ’'Lifetime 
Learning for Physicians” was brought up, and Steve Abrahamson had a very interesting 
comment to make. He said, ’’Maybe one of the problems with the university without walls 
was that it was self-instruction without self-evaluation.” It is an interesting point. 

Steve went on to suggest that self-assessment means regerding patient care, not 
just self-assessment with regard to knowledge. He was concerned with patient care. 

This brought up the question of hospital audit and how to define the pattern of 
quality of care by which a clinical entity at a given hospital will be defined. 1 know 
Clem Brown feels rather strongly that it Is important for the hospital staff to be in- 
volved in defining the criteria for defining quality care. It is important that the 
staff be involved in it partly because that is part of the education. 

With regard to techniques for meeting the needs, there were brief comments that the 
instructor should be able to identify with the community, and it was pointed out many 
instructors do not do this. 

There was a question about the status of the instructor. How important is the senior 
Instructor versus the young man who knows his onions? - 

Also, there v.kS a very Interesting discussion regarding developing audiovisual 
capabilities. I want to point out that Pittsburgh is using audiovisual instruments in 
the Ob-Gyn instruction program. 

I would like to have others from our group comment, 



^ ^^''yer, Bernard: Lifetime Lear nin g for Physicians . Joint Study Committee in 
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DR. CHEZ: I would just like to compliment you on the job you have done. I am 

totally in awe of anybody who can summarize that much conversation. 

I 

KR. HANSEN: I think you did a good job, 

CHAIRMAN RISING- Quit complimenting him and argue with him. You know, Bill, if 
they really agree with you this much I marvel at the fantastic power you have over them! 

Thank you very much, Bill. 

The Consultant for Group Tv;o was Clem Brown, 

DR, CLEMENT BROWN: I will turn our report over to Paul Cudmore, 

DR, CUDMORE: Thank you very much, Clem, 

Half-way through yesterday afternoon Clem announced that he was not making the re- 
port. Somehow I find myself here -- wondering why. 

1 do want to give you our conversation under three headings. Clem's position paper 
was on Educator Obstacles, or Educator Problems, but also I am going to talk briefly 
about the practitioner problems, and then I am going to make a comment or two on insti- 
tutional problems. I will start with the one that wa-^ our prime responsibility, the 
educator , 

I am going to put a few words on the blackboard because these seemed to be important 
words : 



A. EDUCATOR OBSTACLES 

Priori t ies . Really I think our group challenged the group and challenges you. 

W’e are, as you know, in the business -- how much are we doing? We are saying quite a 
lot. Personal priorities of educators was our top thing. 

Then, uncertainity about results. Here, supposing we evaluate, supposing we irr.prov-e 
our educational system, what evidence do we have that this. In fact, changes the longev- 
ity or the well-being of the population? Our group had some real doubts about this. 

Is there evidence that will help us resolve some of our own uncertainities of the re- 
sults of our process in any way? 

Fund 1 ng : As educators we have a measure of control over f 5 nances. How much have 

we devoted to this aspect? 

There is another aspect to funding and this, again, is one of the priiiie concerns of 
our group. If I have a program and I have brought in staff and I have invested a lot of 
ir.y own personal time and effort, there is a bit of reluctance, tlicre is an obstacle in 
me to do soti 1-searchlng evaluation which may, in fact, not justify the funding, or not 
justify my expenditure of time, I think we have to be honest about this and say that 
all of us are probably subject to this negative factor or' tills obstacle to doing honest 
evaluation with that vague uncertainty vdtMn ourselves that our funding may collapse, 
our program may collapse, and all the people we have brought in to work wic’- us may be 
essentially unem.ployed. 

Responsibilities . Wl;at the group got at here, was that the person v;ith the respon- 
sibility, the educator vho has the responoibili ty for evaluation, may have little in- 
fluence on the outcome of the program. In other words, he is not the director, he has 
not the authority to change. For those in our group who are evaluators but not program 
directors, there may be & communicatioris problem around this. 
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Money , I think I may have mixed these two a minute ago, 1 am now referring to 
money, the fact that we have not allocated enough of out own budget for evaluation, for 
this obviously points at educators who do not have the attitude that we talk about. 

M odels . Educators have an obstacle in lack of a good model. There are not good 
models to which those of us who ai*e in the field can quickly turn for help* Our group 
felt this, again, was one of the priority items, and that it would help all of us if 
there were real good evaluation models and these models were readily available; in other 
words, the next thing is the literature. Another obstacle to the educator is that there 
is not helpful literature, so someone must do some iiiodeis and write it up, 

Skills . The last point that I wish to make here is the lack of skills on the part 
of ourcelves as educators. Some of us lack the skill to conduct competent evaluations. 

I think others in the group felt this was not such a bi^ factor, I do not know, maybe 
we hide behind that one. 

That pretty ^-’ell finishes the list of educator obstacles, 

B. PRACTITIONER OBSTACLES 

At titudes of Practitioners , In spite of, or with the full knowledge of Neal 
Vanselow's paper yesterday supporting a lot of positive attitudes on the part of practi- 
tioners, v?e felt there was evidence of a negative attitude on the part of practitioners 
toward evaluation. We felt this stemmed from medical school, and all the other schools 
that we have attended where evaluation is considered almost a weapon of the faculty and 
an obstacle to whoso of us trying to get ahead. Somehow, on this concept of evaluation, 
when you talk to a physician you imiTi^diately stir up anxiety and a bit of i negative 
attitude toward the whole process. We have to deal with this, as i say, in spite of the 
evidence from the Michigan survey that said the fellows were in favor of evaluation, or 
at least it can be implied fron\ that. It is one thing to say you are in favor of it, 
but you know we say we are in favor, as educators, of a lot ov: things in evaluation and 
yet there are a lot of things here on which we are not taking any accion. To say that 
the physicia; is in favor of evaluation is o.ie thing, but to actually have him involved 
in it is another. I had a person,il experience here and I brought tnis up to the group 
yesterday; we have found a degree of reluctance on the pact of ph^'siciaus to enter into 
really soul-searching assessment of what they are doing. 

Educational N eeds . There is too little information getting through to physicians. 

We may be talking about this, but there is not very much information getting Ibroueh to 
physicians and others that suggests or supports the concept tl at an evaluation of their 
educational needs is the beginning of an educatjonal process which is efficient, and that 
the efficiency of the educational process really demands at least the beginning of an 
assessment of needs* We are not selling this concept veiy veil if \;e are going to get 
at the attitudes and get to the practitioner, we have to promote this concept a little 
more. Until that time physicians will be a bit reluctant, 

C. INSTITUTIONAL PROBLEMS 

Lack of Money , Institutionally there is a lack of money, our medical schools, 
our various organizations across the country, federal funding and everytb“’ng, there is 
Just not enough money being focused on continuing education to de^^elop t’ne tiodels and to 
do the sort of things that have to be done with e«/aluation. 

The Syste m. I mertioned this briefly under educator problems, but 1 have to come 
back to it under the institutional problems. Our system is sucli that even when a prob- 
lem is identified, the system does net quickly incorporate corrective action, or begin 
corrective accion, 
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We heard yesterday of tlie American Heart Association’s program on heart sounds. 

This is juet one sinpJe example of this very institutional problem or obstacle in the 
present system within institutions. The same course continues to be given the same v 7 ay 
it was given in spite of the evidence from competent evaluation that it does not work. 

You know there is something in our whole institutional organization that does not take 
note of the results that we produce when wt do evaluate. 

Someone mentioned in the meeting yesterday that there was no over-all institutional 
strategy. 1 believe that sums up this. There isn't any strategy in most of our insti- 
tutions to implement change when evaluation suggests a change should be made. 

Geography ■ Patient care evaluation, which is the kind of thing that we all hope 
to see happen — although some of us recognize that intermediate steps may have some rel- 
evance — Is a geographic problem for those medical schools and other programs that have 
a national audience and also a type of local teaching organization. They are set up in 
Kansas in one area and it is extremely difficult to do patient care research across the 
length and breadth of this nation, and I would like to include Canada in that, as I hap- 
pen to come from there. But the geography of evaluation is a tough institutional problem 
at which we must look. 

Medical Records . The phrase that was brought out in the group was the non-system 
of medical records. If we are to do patient research, then someone has to develop a 
system. There have been attempts. In most of our institutions — medical schools, 
hospitals, offices -- you would really have to buy the deal that there is a ’’non-system, " 
That Is one of the big problems into which we must look. 

We have identified the obstacles. We did want to look quickly at some solutions. 
Call a spade a spade, it is the attitude of the educator. We have to charge or this 
whole thing will not go anywhere. That is the first priority. As educators we must put 
our money where our mouth is. 

The practitioners, their attitudes, here the medical school has a role, and I apol- 
ogize to those of r"Ou who are not physicians. 1 tend to keep thinking medicine. I hope 
you can bridge the gap with which 1 seem to have trouble. We have to introduce or we 
have to emphasize learning exams. Those of us in continuing education must stand up and 
shout loudly that the undergraduate medical school bring new emphasis on learning exais 
and the concept that you learn efficiently when you Identify your own weakne;^ses. This 
is where it all starts. You set your personal objectives on the basis of an analysis of 
your weaknessers, and exams then are a helpful positive tool in learning, not a negative 
hurdle or an obstacle r 1 hope this can come from this group to all of our schools. 
Emphasize the positive educational value of learning exams. Create the attitude within 
the student that evaluation is a tool toward life-long learning. 

Institutions the*' have some expertise in sotr.e area must be encouraged to develop 
that theme beyond where it is now and to let the rest of us know. That is going to re- 
quire funds, and 1 hope somewhere, presumably at a high level of federal support, that 
the kind of money can come that will allow this sort of model to develop within some of 
the institutions that now have experts in the field. 

This has been only the highlights of all we talked over. Clem, I now throw it back 
to you to conduct the balance of our time, 

DR, CLEMENT BROhU : I would like some comments from others In our group. 

DR, EISELE: Clem, Paul did a remarkable summary Job. 
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One of the things that concerns me more than ocher people in our group is the fear 
of the restrictive effects of the demands for evaluation. I fear the demand for evalu- 
ation will force the educational efforts Into molds which lend themselves to evaluation 
but which may not necessarily be relevant or pertinent to actual practice, and we will 
put the emphafiis on some conditions which are not very common that are easy to evaluate 
rather than cn some Important things which cannot be so r'jadily evaluated. 

Someone mentioned the heart sounds training whicVi in six months evaporated because 
It was not useful or was not used. 

Also, someone mentioned a course teaching people how to diagnose pernicious anemia. 
This Is a rare condition, and certainly relative to iron deficiency anemia, it is very 
rare. The best way to diagnose the cause ol iron deficiency anemia is by doing exami- 
nations of stool specimens for blood. This is a very unpopular kind of a test In any 
hospital I have ever been associated with. The importance of iron deficiency anemia is 
many times as important as pernicious anemia. 

The rigid demand for definitive evaluation may actually restrict the relevant <cind 
of continuing postgraduate education, 

DR. THOMAS BROl'/N : I think I can share Wes’s concern that although we definitely do 

have to develop our models, I do not think we can look exclusively at the cognitive end 
of evaluation. I do believe that ve have to make sure that we build into the evaluating 
models the affective end of education. But my concern is that we take a total look at 
evaluation, not just an isolated look. 

DR. CLEMENT BROWN: Paul did a fine job. He really did cover most of what we had 

to say. 

Under educator problems, our problems, you listed '’skills" at the bottom, a.id I 
thought there was more concern with that in the group 

I ha/e some troublesome data, for whatever they're worth, from yesterday that says 
40 out of 47 responding said that success of CME programs must be mtsasured in terms of 
improved patient care, and 38 out of 40 said, "Our medical staff and I v-ould like to 
achieve the skills necessary to extend our CME program on patient care needs and defi- 
cits." 



Perhaps you do not really believe that, but it does not seem sensible that som.eone 
would say "yes" and not honestly feel there is a problem, I chink this is a substantial 
problem. 

The "educationalists" here -- everyone here is an "educator" — the educationists 
here know this, but for those of you who are not educationists there is a delightful 
little book titled, "Developing Attitudes Toward Learning" and its is written by Robert 
Mager. WTiat I wrote in my paper is somewhat stolen from this. To improve attitudes 
toward something, and here we are speaking of continuing education, he points out that 
you Lave to improve the conditions surrounding evaluation, you have tc Improve the con- 
sequences of evaluation, and you have to have some, models of evaluation. That is the 
subject under question. 

As I re-read my paper, and looked at the conditions surrounding the consequences 
following evaluation that are aversive, that would tend to decrease evaluating activity, 
there are some very strong on:>8, pretty unpleasant kinds of things, like you evaluate 
your course and find out there is no behavioral change. That is a pretty tough rne co 
swallow. Or, if you evaluate your course, and study its effects on the learners, and it 
comes out that the learners have not changed at all, even by way of a paper and pencil 
exam or in terms of their quality of care. That is an unpleasant kind of thing. 

One of the little deathless quotes that Mager has in here is, "things surrounded 
s are seldom surrounded by people." 
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There are a loc of outcomes of evaluation that are pretty unpleasant. One of the 
most astounding is: if we evaluate what we are doing we may recognize that our learning 
experiences are totally inappropriate to the objectives in the first place, and it might 
suggest to some of us that we need to do something about how people learn, all of which 

is a shaky kind of concern. There are many good reasons why people do not do any 

evaluation, and 1 am sort of iikeptical about how much of it is going to be done. However, 

I do think it is very important. 

DR. LEMON: It Is Interesting that we do not want to bring in physicians and teach 

them to do something that they are not going to be able to practice. From some of the 
things you have said, and some of the things you are planning, you apparently are talk- 
ing in terms of our taking the time to leatn skills for evaluation which we are not going 
to practice until we get some money. I do think we ought to keep that in mind. 

CHAIRMAN RISING: Neal Vanselow asked if we wanted to go on to Group Five, since 
we are on altitudes and his group discussed attitudes. It seems like a reasonable sug- 
gestion it we are going to be flexible, and be able to modify our^’elves according to the 

way things are evaluated. Does anyone think that it is not ar. appropriate move? In this 

case, then. Group Five will have the stand. 

DR. VANSELOW: I am not going to talk very long because 1 talked long enough yester- 
day, and we have divided it up so a number of other people in the group will help to 
present our discussion. 

Group Five was given the task of discussing the practicing physician's attitudes to- 
ward evaluation. 1 would like to mention a little about the composition of the group as 
one thing. We did not have in the gtoup anyone who could represent a practicing physician, 
so, whatever we say, you might keep that in mind. There were twelve to tour teen people, 
half of us were M.D, 's aud about half were non-M D.'s, again, no practicing physicians. 

There were a lot of people from RHP represented in the group, five or six 1 believe, 

kliat particular point of view they would take, 1 am not sure. I would not know into 
which category to put them. 

We decided that 1 would make a few introductory remarks, and chan ask Frank Lemon 
ard Jim Cooie to report on the meat of the discussion. 

The first point 1 would like to make is in rebuttal to Paul Cudmore, If you look 
at my position paper the point I was trying to make was that physicians are willing to 
accept the relatively superficial types of evaluation, but if you look at the data again 
you will see that what most educators- consider to be real evaluation, that is patient 
care evaluation, practicing physicians are not very willing to accept. I think there 
are ma'iy , many concerns practicing physicians have about this level of evaluation. 

There are two points 1 would like to make. The first is the matter of simulation 
of the patient care situatior.. This was brought up early in the discussion and pointad 
out that Steve Abrahamson had Sim One type of model. Also, there have been a lot of 
paper slrauiitlon techniques which have been used recently; the pencil and paper type of 
thing with erasures and branching patient care problems. The question was, "would this 
type of technique be more acceptable to tlie practicing physician chan actually going in 
and looking at his records?" Most people think it would = If there were a correlation 
betweet; ?ncll and paper simulation and vhat the physician actually did in practice, this 
would be one way around this particular problem. 

Someone mentioned there was a very high correlation between pencil and paper simu- 
lation and actual audits of records. A few of us were net quite sure of that, and wlien 
we get to the discussion later 1 would be Interested if anyone actually has some data. 
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The other thing I wish to say is that the one thing the group could agree on was 
that we are having a very diificult time selling meaningful evaluation to practicing phy- 
sicians, We spent an hour discussing this, then an additional hour proposing some pos- 
sible solutions to the problem, 

I am going to ask Frank Lemon to present the first part of it, and then ask Jim 
Coole to present the second part. 

DR. LEMON: One thing that did not come through but should have is that I feel like 

I am representing practicing physicians. My background includes general practice in a 
rural area of Wyoming, and, more recently, the practice of general Internal medicine 
in southern California. 

I have been bemused as we've talked, that what is really needed Is a controlled 
study on whether it makes any difference to go to any doctor. Obviously that has impli- 
cations for continuing education because if it does not make any difference continuing 
education does not make much difference, and evaluating it would not make any difference 
at all, 

I think our group in general would say that physicians' attitude toward continuing 
education evaluation is unhealthy. 

If physicians are considerably apathetic toward current continuing education efforts, 
their attitudes toward evaluation of themselves as an essential component for gauging 
educational needs and for designing programs, are downright suspicious, resistant, and 
in many instances potentially hostile. That is not a very cozy environment for the con- 
tinuing education salesman and evaluator. It might even set the scene for another 
"Death of a Salesman." There is no time for an ineffective soft-sell approach, and a 
hard one looks as though it might be hazardous. 

As Dr. Vanselow has said, we first agreed that continuing education evaluation does 
not usually sell well, at least the effective forms of it. Then we ask ourselves. Why? 

The reasons we came up with certainly cannot be all of them, but we thought strongly 
that there were two major reasons. 

First, the areas in which there was strong agreement could be summed up, again, in 
one word — it is a threatening enterprise. Threatening, how? Some saw It on behalf of 
the doctor as threatening to his own self-image. I spoke about the mornings when I felt 
engaged In insanity, and that is an uncomfortable morning. No doctor in practice likes 
to feel uncomfortable in what he is doing that he does net really know the significance 
of, and is not fully in control of. Evaluation may focus attention on that as a fact of 
life. 



Others saw this threatening situation as his in-buHt bias against evaluators, some- 
what related to what Dr. Cudmore spoke of a few minutes ago. Of course, that might be 
evaluators in general, it might be evaluators specifically from certain agencies or 
certain enterprises. 



In that connection it was felt there is a rub-off to the continuing education enter- 
prise and to the evaluator in that field of the well-known "town and gown" syndrome. It 
may very well be that in this area ve are getting some of his vote of "no confidence" in 
the university’s perception and understanding of the problems of private practice. Ihey 
really are two different worlds. I can appreciate this better than many others in the 
room because I have had my feet solidly planted at different times in both worlds. 



Others felt it was threatening because there was a positive fear of the misuse of 
data so collected by people right in his own community and his own his peers, 

not to mention the misuse of data that is potential, ynd for which iuctances can be cited 
of the misuse of such data by 'outsiders.' This reflects to some extent a doubt about 
the confidentiality which we may try to assure and the anonymity vhldi ve may promise. 
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Ihere was a feat expressed by others that he feels this may somehow be involved in 
the growing process of "the control of medicine ” It may be so used. 

One could sum up this threatening thing by saying that in an era in which doctors 
are under attack --- and lei. us not forget that chat is the environment in which we now 
operate — it is not a very propitious time for going around auditing what he is doing 
In the hospital and in his own private office 

Dr. Cudmore brought out what we thought was a second very important area in our 
group. I’ll say it in different words, but it means the same thing, That is, evaluation 
of the type that we are interested in is not a part of his undergraduate experience. He 
has not been trained in this as a way of life, self-evaluation for the purpose of self- 
maturation, self-growth, identification of needs, etc. Undergraduate examinations are 
feared, detested and, as all of us know, are even Inaccurate. failures in the under- 
graduate system mark the individual as a failure, but do not mark the system and the 
evaluators and the educators as failures. Therefore, we come along and talk about eval- 
uation, and the purpose of our evaluation is nou understood because it lias not been a 
part of the way of life. Perhaps we are unsuccessful and do an Inadequate job of trying 
to indoctrinate him as to the purpose of our evaluation. 

He does not believe that the evaluation is necessary in many cases. This is another 
reason given. He does not believe that you and I — particularly those of us from the 
university who are there because we do not know how to practice medicine, and most partic- 
ularly those of you who are not educators in medicine at all — can have any under- 
standing of the art of medicine, and could not possibly evaluate something which will 
not yield itself to numerical values, things that Involve intuition, patient relation- 
ships, judgment, the balancing of pros and cons and pluses and minuses. How are you 
going to measure these things? 

Finally, he does not think much of evaluation, probably because there has been in- 
adequate feedback or no feedback in his past experience from that evaluation which 
could affect his thinking and planning. One more thing, when you are talking to him 
about evaluation and utilizing his records, he is thinking, ”How can my records be 
compared with anyone else's down the street or over the state?" 

There were some miscellaneous reasons, other than these two or three major ones, 
which were given and which we thought ought to be given more thought. 

Evaluation is time-consuming for the physician, and thus annoying. There was dis- 
agreement in our group about this, but it was interesting to me that the disagreement 
seemed to come mostly from people who had not been In medicine and who argued that if 
you could find time for continuing education you could find time for evaluation, I do 
not disagree with that, but in this day and age one simply has to realize what a private 
practitioner's life is, and what his feeling about paper work is in order to appreciate 
the significance of anything that consumes his time in paper work, 

Another miscellaneous reason given was cost. This was looked at from two view- 
points by those of us who considered it. In some Instances the fancier evaluation ef- 
forts cost to the physician — in terms of both money and time -- and cost otherwise to 
the various agencies which might be mounting these kinds of efforts when it Is recognized 
that these do cost, and as a citizen and taxpayer, the physician wonders who is footing 
the bill. 

1 suppose it would be only natural, since our own ego status is involved, that we 
seem to put off until last the reason — the real reason ~~ why evaluation is not suc- 
cessful. Maybe we are just poor salesmen. This should have been first, as I have said, 
but we have put it last. 
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Dr. Petersen raised the question yeste.day very vigorously, having first said that 
behavior change is the only thing that is worth evaluating. If behavior change is the 
thing to do, of what are we fearful? Why don’t we get busy and do what needs to be done? 
The answer from our group seems to be that we are fearful of the physician's attitude. 

MR COOJ 4 E: It is rather an awesome task to follow a presentation of problems with 

the presentation of recommendations, especially when our group brought up two rather 
negative means of assuring evaluation of a relatively stringent form. 

The first of these was the suggestion jhat we need something in the way of mandatory 
re-examination of physician competency. 

Another idea was the task of trying to sell to third-party payors and the public 
the need for physician competency, and by exerting pressure tlirough these two mechanisms 
it would force the physician to submit to self-evaluation, and outside evaluation. 

There was no real consensus on either of these items within the group, and if any- 
thing could be said as a result of the discussions it was that the group felt that some 

periodic review of a physician’s competency was desirable, but, on the question of 
whether or not this should be mandatory and whether or not this should have any punitive 
features there was no agreement. There was a relatively heated discussion for quite 
some time about this one point. 

On the positive side, however, I think we did come up with some ideas that bear 
further examination. The one that 1 personally feel needs to be promoted the most is 
the necessity to expose medical students very early In their training to the idea ot 
self-evaluation. It is necessary to promote with these phys icians- to-be the attitudes 
necessary to carry them through a life of self-learning, self-assessment, and self- 
evaluation, which would make these other negative aspects unnecessary. 

In order to do these things we need much more data on physician attitudes about 
continuing education and the evaluation of this education, and the reasons that lie be- 
hind these attitudes. Ve have a kind of visceral feeling that physicians are antagonis- 
tic toward continuing education, or antagonistic toward evaluation and evaluators. They 
give us this impression. We feel this is true, but we have no solid data to back this 
up. What is more, we have no data to Indicate why these attitudes do exist 

The real core recommendation that the group seemed to feel lay behind all of this 
was the necessity to involve th.e practicing physician in the management and evaluation 
of continuing medical education. We need to sell the idea to the physician that he has 
to take an active part in both planning and evaluating any program of contii:uing medical 
education, be it from a Journal club to a formal postgraduate course attended by people 
from all over the country. 



Also, we realized that we need to educate the educator. We reed tc present to him 
the aspects in education that would enable him to be a mere efficient educator and 
evaluator. It is an old saw in education that those who cannot teach become principals; 
and those who cannot be principals go to college and teach education. I think this is 
becoming true in many fields. Dr. Frank Lemon alluded to the idea that physicians 
think that those who cannot practice medicine go and teach medicine. 



There also seems to be a feeling that if you are an outstanding scholar, if you 
know the field very well, you are automatically a good teacher. Being an educator, 1 
cannot accept this. It is time we realized that many of our physicians who are in the 
field of education are good physicians, they know medicine very well, but tliey are poor 
teachers. Let us give the skills to be good teachers, and maybe this will change some 
of the attitudes that physicians have about continuing education and the evaluation of 
that continuing education. 
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We also need to convince hospital administrators to accept the responsibility for 
implementing continuing medical education for their staffs and this implementation needs 
to be in the form of verbal support and monetary support. We need to convince these 
administrators to do part of the selling job for us. We can do this by showing them 
the benefits that can be derived from a solid continuing medical education program. 

There needs to be a greater relevancy of evaluation to the patient care that a 
physician offers. 

We have all done the happinei^s scale, the popularity thing. We got a little deeper 
in pre- and post-testing. Frequently we have done this and never told the physician 
what the results were. I think, one of the important things in any evaluation of his 
continuing education is that we tell him what we found out about what he has been doing, 
but many of us never do this. 

We also felt that the area of needs assessment is neglected more frequently than 
it should be. 

Finally, there needs to be some form of in-service training foe evaluators to keep 
them up to date and on their toes about the use of the data that they collect, and the 
necessity for maintaining relevancy in the educational program in which they are Involved. 

I trust that any of the rest of the group that has any disagreement with my summar- 
ization will at this time let themselves be heard. 

DR. VAIJSELOW: Does anyone else in the group feel moved to comment? 

MR. JENKINS; Neal, a comment that you made concerning the pencil and paper tests 
to replace audits; I think I heard you say that we felt they could do that. 

DR. VANSELOWs I think there were several people in the group, or at least one I 

can remember, who thought there was a high correlation and there were many who thought 
there was not . 

MR. JENKINS: As 1 recall, Jim Coole was the one who brought this up. There have 

been a couple of studies but they were fairly selective, there was no real randomization 
to show ^ny correlation that it can, that there may be a possibility, but there was not 
adequate proof at this time for us to make that statement, that the paper and pencil 
test could replace audits. 

I am not sure that we came to the conclusion that this is not the time to do audits, 
because of the conflict that everyone seems to be under the pressure. At least 1. as an 
individual, would feel that despite the pressures we are under from all around, we can- 
net put these tasks off, that now is as good a time as any. We may never reach the time 

when auditing would be an acceptable practice, so we have to show how it can do the 

job and sell it as an acceptable tool, and then go about it. I do not think we can 
continue to wait just because someone has some anxiety. I think we have to attack the 
problem , 

DR. VANSELOW: Any other comments? 

I think one point I might make is that the sense of 'irgency in this business came 
up yesterday, che fact that people are considering recertification and relicensuie, and 
Che federal government may be getting into this. We Co not have a lot of time to waste 
getting some sort of a system of evaluation of what we are now doing into effect, 

CHAIRMAN RISING; Thank you. 
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All three of the group reports so far have had reference to money: One group called 

it ''funding/' another spoke of it as "costs/' and the word "money" was never actually 
used. It 7as referred to both as a problem and as a solution. It is quite appropriate 
that we did have those three groups present tandem, and it is even more appropriate new 
for Bill Herzog's group to report; Their discussion was centered, at least originallv, 
upon costs of evaluation. 

MR. HERZOG: Thank you. Everyone was in a good mood yesterday. We I'ad a lot of 
discussion, no real debates ^ Maybe we will get some today 1 think 1 will introduce 
John Barson from Office of Medical Education, School of Medicine, Michigan State 
University, who will make our report. 

DR. BARSON: What the group attempted to do in examining the cost of evaluations is 

essentially arrive at some rational resolution of the Issues that are involved rather 
than trying to establish technique, or even give priority to methods of costing. Three 
major issues came up with regard to cost factors in evaluating continuing medical edu- 
cation. 

The first question ought to be the first one applied to a program, "Can we afford 
not to evaluate continuing medical education?" There, curiously enough, are two answers 
that seemed to come up in our group to that point — one is "Yes" and the other is "No," 
depending on the viewpoint and who gains and who loses. 

There are some areas of learning that apparently are not significant enough to war- 
rant the expenditure of funds, or upon which the specific measurement of gain is not 
critical . 

A friend of mine used to ask, "How thin do you want to slica the boloney anyway?" 
This, in effect, is the gist of the decision. If there are high risks involved in not 
evaluating, the cost can appreciably climb, and justifiably so On the other hand, where 
there is low risk involved in not evaluating, it may be well not to burden the program 
with additional evaluation costs over minimal measures 

Another area of cost, identification of costs, and counting all costs has to do 
with the cost of ineffective evaluation measures. Inappt opr iate or Inefficient evalua- 

tion measures obviously are more expensive than the effective ones, and yet quite often 
these costs either are ignored or somewhat concealed iu the cost of teaching. Admittedly, 
ineffective or inefficient evaluation measures can double the costs which were cited 
yesterday by Bill Herzog, since we not only lose what it costs us to do the evaluation 
but also the losses of not evaluating appropriately are also added, 

A third area of counting all the costs has to do w^ith differentiating between the 
development costs of evaluation end the application costs of evaluation. If you ask any 
manufacturer or producer of material goods you will find there is an enormous amount 
of Investment in research and development, the R and D factor. Research is often 
identified easily by laboratories and other sites and types of workers. The development 
is not quite so easily identified. The truth is that most educational institutions 
spend the bulk of their budget in development and a minor portion of their budget in 
delivery of what they have developed, but for some reason they are unable to articulate 
the fact they are developing instruction, they are developing evaluation, and these costs 
have never been identified. The models are seriously lacking. 

This is my personal experience, the ratios 1 have seen vjith rogard to development 
coste versus application costs in teaching-learning situations ^:ivwhere from 4:1 

to 11:1, which are very high ratios. This underscores the expense of ineftective or in- 
efficient evaluation measures- 




A fourth point an counting all the costs has to do with considering the amortization 
of costis from repeated use of proven evaluation measu;os> That is to say, the more often 
you use something, the less cost it is to you, especially in the case of evaluation where 
che::e should not be too much wear and tear, short of having to alter the program to met 
minor changes in the person! ei who are being evaluat'^, or for certain factors, Tl\ir 
amoi t izat ion factor is very seldom c: ted in account in,.; procedures, but proven methods 
tan have a lower and lower cost as they ate used more «nd more widexy. 

The fifth point under counting all the costs liuS to do w^ith relating the evaiuat Uni 

costs CO the available rescurres Tor the pt ‘^gram and ictuai total progra'j cost, v%Tiat 
seems like an exorbitant amount for evaluation Tigitt iiv.t seem so much if we were lc 
the actual expanse of a program which is bevig sponsored, counting all the costs r. laf 
to it- This issue of relationship was brought up by a number of speakers yesterday, 
ic wai: r.ot explored much beyond that because we would have to get to specifics at thac 
part, but this is certainly a factor chat \:e would have to take into account in coutitina 
the coscs> 

The summary of counting all the costs and the rationalizing of whether it is appro- 
priate or not to expend funds in behalf of evaluation has to do wich the question whttliot 
wc; are trying to do better what should not be done at all. This issue came up rep^^ edly 

in the discussion, that we may really be trying to improve on something which should be 

reduced- Again, thi-i is another factor in cost accounting for which we must account - 

The second major issue of evaluation costs, cut}0UF.ly enough, does not look Like a 
cost i.isue at nil, as has been cited earlier in the presentation today, and that is the 
use of medical practice as an efficient base on which to establish education evaluation; 
for instance, the medical audit: 

Several points were made with regard to, "'How do we select an appropriate evaluation 
measure?” Under that the first point might be that evaluation is app’^oprlate if it is 
a part of his continuing education it probably should be closeiy allied with his practice 
ot medicine. This suggests that the use of .nedicai practice as an efficient base is 
ju.stified. At least this was the consensus of the group. 

The use of medical practice as a base for evaluation has several legal and ethical 
considerations- We are striving fer efficiency, but the question is whether the cost 
or efficiency, so to speak, car, be paid by people :n the profession, or w’hether they 
are willing to pay it. 

A third major issue with regard to ccsting evaluation has lo do with the selection 
ot the evaluation method itself, and tliree points were made under iliai by various members 
of the group. 

First, and this sounds terribly obvious jut an iinpoctant issue to keep in mind, 
that the appropriate method is probably the beast expensive over tl,e long term, 
sheer logic of that stuns you the appropriate method of evaluation Js tlie least ex- 
pensive over the long pull. Inappropr iat*. methods, regardless of how inexpensive they 
are, certainly are not worthwhile. Just .he garden-variety shopper knows this sort of 
tning, but quite often it Is ignored in ectablishing evaluation programs. 

A second point under selecting evaluation methods Lax to do with knowledge of cost 
estimates for evaluation, and these cost estimates are feasible and deplrable, according 
to members of the group. Several illustrations of cost estimates were cited, one of 
them being one-third development costs, one-third application costs or instructional 
costs, one-third of the total budget applied to evaluation- This sort of ratio appeared 
in several projects mentioned yesterday There may be other ratios These are also 
variable costs, depending on how much advanced information we have about learners and 
how finely we wish to evaluate the experience that they have had in learning. 
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A final point in selecting evaluation methods has to do with a discussion held in 
the three sessions preceding this one, and that is the coordination of evaluation ef- 
forts V7ith data collection regarding medical practice, the proposal being that evaluation 
would cost less if it could be a part of the cn-going process of selecting data on 
medical practice so that we would not have lo construct criteria and other measures to 
apply to certain learning situations. The results of medical care would be available 
to support or to change programs that preoentiy are in effect. 
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point has to do with the information exchange among evaluators 
to establish effectiveness guidelines among alternative rreans 
person approaching the evaluation dCLision is not working in 



I would like to cite the one model I was responsible for introducing yescerdav in 
summarv: It is that in a sense the evaluation process that educators engage in, the 

decision-making, is not too unlike the diagnosis of a physician in working with a patient 



The physician has a certain amount of irfotmaCion about the patient vv en he enters 
his office. The more information, of course, the more sccutate the diagnosis could he. 
In seeking further information regarding the patient, his condition — m aLtemptlng to 
decide what sort of treatment to apply to the patient — he has a certain number of con- 
straints: economic constraints, logistics, whatever other problems he may eni ountar. 

He might order $1,000 worth of tests on a person to gain every bit of information he 
can, or he might choose to employ a much simpJer approach, depending on the resources, 
and estimate of the seriousness of the problem. It is a decision-making process that 
the educator also faces. If the physician is able to deliver adequate medical care, 
with the large number of patients he is dealing with, there is no reason why educators 
could net make similar decisions without, as I said earlier, slicing the boJon^v 
thinner than, necessary- 



It \s very difficult at this juncture of the discussion to say that 
all the points mentioned, I think Bill wishes to open the discussion to 
group members who will kind of gloss the tDp of some icebei gs we touched 
cussion 



I i.avn covered 
the rest of the 
on in our dis- 



MR. HERZOG: Thank you, Johric Are there other poincs that others in the group 

might have about the discussion? 

DK PERLMAN: I thought you covered our nany , many points very well, John There 

were two that you sort of passed through, and 1 would like to re-emphasize them because 
they do hrve an effect on ccsls 

The first one is that v;e calk quite a bit about defining evaluation and defining 
what we should be costing. One of the things that several of us brought out was Lb it 
you have to decide what you ace trying to prove or what you are trying to do with t alu- 
atlons. if you go at it as industry may, or some type of very scientific research ;hat 
wants to prove a definite point absolutely, we may find that it is very expensive to 
evaluate and we may never get there. Instead, if we can be satisfied to show some 
correlations and relationships between a particular educational program tnrough an eval- 
uation of it and seemingly some change in behavior, this may have to be satisfactory 
for evaluation, because of the many variables that we Cc.nnot control. 



Tying onto that, Bill, you are really skipping something that you brought out, re- 
membering Bill’s paper from the other morning, all the costs and the very differ.'=int per- 
centager of costs for evaluation. Talking about what Hugh Petersen said about using 
evaluatioii of behavior as our major step, we felt we should not throw out all the other 
types of evaluation, e en If we agreed that evaluation of a physician’s practice will 
give ufi the best a^'wer . What we should do if have several examples oi several studies 
that try to match tind try to evaluate the metliodu of evaluation so that everyone is not 
O survey which costs — what was it? -- 25i) percent ot the cost of the project. 
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whtLh 1 think is happening now. There are many sucv-’eys, and a great deal of money is 
going into them, Perhaps several groups with a very spetiilc feasibility study idea 
eight to be doing surveys, versus clinic areas, versus practice to see if there are cor^ 
relations. This is a little like what one of the earlier groups said about evaluating 
simulation versus practice. 1 think these two things are extensions of some of the things 
you had to say. 

MR HERZOG: Are there any other points that were brought jp that others in our 

grou; would like to mention' 

CHAIRMAN RISING: 1 was going :o say, if nobody else wants to enter any further 

comment, that I would under score something that. John said. It really makes a difference 
what you at 3 evaluating for, what objective you have in doing the avaluation. If you are 
merely evaluating in order to substantiate a request lor more money from some granting 
agency it is too expensive even if it does not cost much. If you arv^ evaluating as part 
of the educational effort in order to plan your program, in order to see how well you 
have done, in order to change course higher costs are justified- 

As Dr. Clem Brown pointed out, if it is for program development, or actually if it 
is a major part of the education and in his type of situation the evalu^’tion Itself 
is the education — the cost of evaluation is easily borne, 

The formula chat was poitited out, one-third, one-thiid, and or.e-thlrd, we got to 
calling the '^pie” formula in our group because the "p*' was for planning, che 'l" was 
for imp lement ation , and the ''e” was for evaluation. Literally both the ’^p" and the 
both the planning and the evaluation are part of evaluation- You evaluate in order to 
plan, you then are justified in spending a fair amount of money. In tevievnng some of 
cur cost figures that 1 have rcc ntly gone over with Bob Keth, I think ic is a ball-park 
figure that we probably spend two-thirds af outr treasure in planning, using evaluation 
of needs and past programs, and evaluating current t^rograms, 

MR HERZOG: 1 tnink there was one other thing we brought out and I do not know 

it we should bring it up because we dii not get near c solution — but in funding and 

costs John mentioned that you have to delineate what the costs aie, and we spent quite 
a bit of time discussing long-range costs versus short-range ^ rbough we did not use 
that term. We may find that one of the methods that costs 250 percent to evaluate will 
make the whole education program cest much less over the next ten years, but tt Is a 
little bai d to convince the funding agencies of tills factor, and f think th.’t afternoon 
we would probably all be pleased to hear solutions- got as fat as saying that that 

was a terrible pioblem, trying to cen’ince them thit some new method or sore more ex- 
pensive method of evaluation is goire to sa’^e money over the next ten or fifteen years 
They are looking at the tax dollars that -ome in this year -- per..ud. 

DR. BARSON: I underscore lack PerimarMs p^int for the fact that your costs can he 

increased by poor operation of other po. t lon.= che program as well, that quite often 
w’e are paying a high price for evaluation when actual iriprovements in the instructional 
program or the planni‘,g phase might reduce those costs, so these figures are hnrd to 
nail down to a firm amount. 

I "am almost und®r the inpreosion that evaluation really should be a veiy minimal 
expenditure, if the preparations for it have been made appropriately in other parts of 
the program, that w^e may be paying a price for Inefficienwy elsewhere, like the evalu- 
ation study. 

MR. HERZOG; Some points were brought out in the group that w^e really did not reacli 
a consensus on One v;as that we probably have not spent enough on evaluation in an in- 
tensive way, so we really do Tot know a Lot about out measurements. We have a lot of 
guesses ahoji the accuracy of the "react ionnaire’' and what it might indicate toward bo- 
havior change. Several of us talked about the desirability of reslly Intensively eval- 
uating, trying to test all the tools and find out about how valuable our tools are be- 
fore we arc willing to cast out this tool or that becaise we do not know enough about 
t ® .8, So, maybe we are not spendl*ig enough, as In so many other areas, 
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Secondly, 1 think that we ought to have more centecs that are particularly looking 
at evaluation, and maybe there ought to be more agreement among those centers as to how 
they were going to specialize so there will be a little more efficiency in their own ap- 
proach toward evaluation. 

CHAIRMAN RISING: Thank you. The summar izat ions have been excellent this morning. 

1 have been very impressed by them, knowing what vent on in Bill Herzog’s group, I 
know that John Barson did a raarvelous job in condensing it, ana I want to thank him. 

The final group report. Group Five, wall be by Hugh Petersen, the purist. 

DR, PETERSEN: A little purity never hurt anyone. 

CHAIRMAN RISING: Just so you don’t overdo it I 1 believe in moderation in every- 

thing, even purity. 

DR PEri:RSEN: Dr. Dora Id Korst will give our report. 

DR KORST; It is interesting to be last cr this program and tc have listened to 
t’r.e varlouj summaries because certain things begjn to come through, and I think as we 
sift this coni.-renco we are going to find that wa do actually have soma consensus of 
opinion . 

Hugh Petersen made a good position yesterday in his wcitcen report on the nends for 
behavioral methods of measurement. Wg; argued the position that the meaningful measures 
oi performance changac Would the behaN'ior of ph>'siclans in applying knowledge gained 
in continuing education ultimately Influence pa; lent carel' The problems lie in how to 
develop the measurements of learning behavior. Wc discussed the log, cal ieiiability 
against the statistical reliability. The question arose as we approached the problem, 
’Vhat ace valid criteria?" Are these mete paper-and-penctl testa, or can we really go 
firtber than this? We debated the behavioral versus the nonbehavior.al goals as the 
best methods for e aluation- We defined six problems 

The first problem was what are the methods that measure change iii performance, be- 
cause a charge in performan^'e should be an improvement, ibis is what wo measure- The 
discussion included pee- and post-test trig , vocabulary tests, auseasment of Individual 
performance, evaluation of l>ospital medical records, problem solving or case simulation, 
results ol patient care, improved physician coniiLiunicat ion, utilization of selection 
patterns c£ laboratory tests, and hospital, utilization patterns. 

in application of any of the measures emnhaais was placed on *he irportance of the 
iiianner that the Methods are applied so that safeguards exlSk, to protect the confidence 
and the trust of the individual physician. This has come out a number of ^imes in tVie 
d Iscuss.lon . 

Tne third problem: the communications between institutions, such things as the 

HAS program of hospital record auditing, anl the medical audit program. .Uso, the prob- 
lem-oriented case record, Lerry Weed’s system, was discussed 

The next problem, the fourth, was the attitudes of physicians to behavioral assess- 
meat£ which ha« been discussed at length this morning. Who should decide the behavior- 
al criteria? We felt thi^^ should be a consensus of the phys.^clans involved. It is very 
important to bring the p’^vslcian himself Into his ovr. assessment and into establishing 
the auditing criteria or the medical center criteria. The success or f ’lure of ^editing 
v^ould probably depend on the confidence of the physician. A functional set of criteria 
would therefore be developed if the men in practice were Intimately involved in this 
process 

The fifth problem discussed was the popularity evaluation C^n we really go beyond 
O we leally evaluate shot" courses? 
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The sixth and last problem that we discusscjd way the motivation Lo participate in 
programs In order to be evaluated, In other words, how does one get a representative 
sample or an unbiased sample, because we are all aw^^re that there are certain physicians 
who regularly attend continuing education, we have strong support, forty or fifty per- 
cent of physicians in some areas, but there is a group that we do not reach, jjo they are 
out of our sample of evaluation 3t this time. What are the attractions to programs? 

How can we stimulate this physician who is out of reach? Would the hospital medical 
audit stimulate the education? 

The Min ority Opinions I think other members of the group may want to debate this, 
because 1 am not sure these were clear minorities. The first point: testing of vo- 

caoulary, pre- and post-testing, the paper-and-pencil rests in general were sort of a 
minority opinion that this was going to be adequate behavioral evaluation in the future. 

I would like to ask other members of the group to make some comments. 

DR, PEThRSEN: I think, out of context, that is going to be a meaningless statement 

regaiding the vocabulary testing because it does not seem to relate to anything. This 
came out of a discussion having to do with validating, using behaviorr.l measures and 
^‘elating them to nonbehavioral measures, and the comment was made chat In the long run 
such a thing as a vocabulary test might be a simple, single, positive, predictive indi- 
cator of later succc' s in som.c kinds of work that the physician would encounter. Fo, it 
w’a^ an example only. 

DR. KORST: 1 think this Is what we wane because I am not sure it is fair to just 

arbitrarily divide these into minority and majority opinion. 

DR. PETEPSEN; I am in the minority all the uay through this. 

CHAIRMAN RISING: Don^t be paranoid, Hugh 

DR. PETERSEN: Wnat have I gJt to fear? 

D'l KORST: ’the second point: the partitioning of continuing education activities 

to suit the individual needs of the physician; in other words, the very specialized con- 
ference. 1 think there was not a majority opinion that this was a good direction to go. 

The third point: is patient care the only goal? Everyone seemed to think it was 

the major goul Ac least we did not have sttong feeling that there should be ether 
goals than patient care. 

The fourth point: tuc popularity — or I think it was well put, the happiness poll 

of programs — not much enthusiasm there. 

The fifth point: this one really should be la., or something In the majority, be- 

cause I do not think vc really decided minority or majority on this, but I think every- 
one felt there was a real need to know more about what indi^fldual departments or offices 
of continuing education are doing. There ought tr be some coordination or some way of 
disseminating the information about what programs are being evaluated, how they are doing, 
to avoid unnecessary repetition in wc^sts. That really should not be under minority. It 
is a separate thing. 

The Ha j c ^ i ty Opinions . 1 really lope our group will discuss these. The first is 

motivat icnal factors, which seem terribly Important, These should be studied further. 

The achievement of improved patient care seemed to come out as a universally accepted 
motivation of pnysicians. The physician responds to this In a very positive fashion. 

Any evaluation programs that were related to the achlevem.ent of improving patient care 
were well' accepted . 
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Second: sample size — and this really follows with motivacicnai lact:<ts — In some 

way we ought to improve our sample size, the disct tbotion or our samples. This is 
reaching to the physician we are not reaching now, and chic is absolutely necessary to 
get a representative group with which to work It we are going to gee any kind of a 
random sampling of impressions and programs, it is necessary to broaden the sair.ple. 

Third — we spent a lot of time t-^ Iking about this; smaller programs related to in- 
dividual physicians or to Individual hospital programs; in other words, going out to 
the community. We felt this was a g^'eat need, to work with individual practices, in- 
dividual hospital programs that involve the hospital staff, the hospital administration 
in a very direct manner, the hospital organization invo Ivcment toward (1) self-evaluation 
and (2) auditing. These two things, the education and the auditing, ought to be bi.ought 
together in the smaller program where the physicians themselves participate. They may 
develop their own criteria, with help, with aids, with suggestions. The universities, 
the continuing educaiJ.on departments can supply the expertise, che advice, the education, 
but if the physicians themselves help In developing che program it should help the moti- 
vation. This ir Dives the case record review, or improvement of case records, this also 
involves the hospital staff time investment and the developing of hospital education 
programs . 

Fourth: some minimum requirements for continuing education. This relates back To 
number three, that the hospital auditing and education committee within its own organi- 
zation can begin to develop a minimum requirement, not a punitive sort of approach, but 
a constructive approach that might point out individuals oi direct individuals into 
certain kinds of continuing education where the hospital felt they had a weakness, through 
auditing or through self-assessment. The standards of records would he required in this 
program . 

Fifth: cooneraMve or coordinated studies should be started. These would be 

selected programs through continuing education departments chat vjould be developed to 
obtain a sample size that would allow random sampling to evaluate effective measures 
of behavioral evaluation as to reliability, success, cost, numbers of measurements, etCc 
The validation of the behavior needs a random study to include the comparison between 
the statistical probability analysis, the paper-pencil tests, with the behavioral eval- 
uation that will affect, ultimately, patient care and hospital utilization, 

W? felt the test program or the study sUou lu be small enough, it should ho -^imple 
enough for a sfugle tcpic-or i ented progLam that could be use.l in a number of centers for 
the evaluation. The physician participation was essential to ge " the physicians’ self- 
acceptancc of a program- 

One example idea that was discussed would be a coronary care program where this 
would be tested as a uniform approach a program, a uniform evaluation. 



Finally, this is my own comment in summary to what we have heard this morning. It 
seems to m. that the time is now. My own assessment of physicians in practice in com- 
munity hospitals Is: they are moving very rapidly in the direction of seif-assessment 

and of auditing their performances. There is a very strong feeling among physicians 
that it they do not do it, somebody else will, I think this is an accepted fact, and 
that you will find a very receptive audience in most hospital staf." organizations. 



DR. PETERSEN: I would like to make a few aaditioral comments, Just to reiterate a 

couple of things. We talked at great length about behavioral measures and there was 
quite a bit of argument about whether or not that was really an appropriate approach. 
Someone would say, "well, does it not depend upon yout objectives^' Since i have been 
around Steve Abtahamson for so long 1 answered that Imuiiediately . My question to you is; 
do we really have only ^ne objective? We only have one objective as far as I am concerned, 
and that Is improving patient care. We do not have an objective that is called ’’amusing 
] ^ in students. ” 
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DR PETERSEN: Again, 1 an naive, I guess, but I have been here as a purist, as 1 

say, if w’e are only trying to do that, if that ^s our goal, then we have to have behav- 
ioral neasures . On': 3 we have decided to do that, we can take a slightly gentler course. 
We can say this: if we can cone up with some definitive studies where nonbehavicra 1 

measures are validated with behavioral measures, where we can show very high positive 
correlation, then we have sortie empirical basis for saying that we do not have to eval- 
uate using behavioral measures every time. The areas, that is, the domains wherein we 
do this, have to bo very well described, very well planned, and definitive studies must 
be made , 

Dr. Korst was talklr.g about sampling, and it Is a very tough topic. Perhaps 1 can 
no\! show you It was related to what were talking about. Supposing we take all of 
the CME programs in coronary care throughout the country. If stratification within 
that domain Is necessary we could do that, but let us just say we take that domain and 
we randomly sample ten of these and re-throw our evaluation efforts into those ten where 
the results can be, because of their statistical characteristics, geneializable ^ then 
the money would bn well spent. Since we cannot use behavioral measures for e'/ery study, 
obviously, this o :her approach is to me the only logical approach at this particular 
time. So, it is a validation study, and the results could be generalized to a much 
greater population than we can at present. This requires great cooperation, and [ am 
not sure where that is going to come. 

As far as the rest of the group is concerned, do we have any comments here? 

DR, WELSH: I want to congratulate Don Korst for bringing such order out of chaos. 

As one of the majority stated, all of the groups have articulated the fact that the 
prime motivational factor for most physicians in practice Is iT«.provement of patient care. 
To echo what Dr, Korst said about hospitals belug ready and moving to develop a system, 
they see the handwriting on the wall. We all see that if they do not do it, someone 
else will do It for us, then actually the setting is ripe for the development of a good 
behavioral study in which they are going to develop their own cciceria, whether a la 
Clem Brown or what other system they may use, they are going to develop their own 
criteria which will threaten them the least, which will b.^ in-house, which will give 
the tools to tViem for improvement of their own patient care activities. They will need 
some help in this, and I feel the role of the ^ducational institution is the resource 
to aid them in accomplishing this, and not to do it tor them- 

CHAIRMAN RISING: Thank you very much. We will have discussions of all the group 

reports this afternoon 1 encourage you during the lunch time to conoideiT what you 
want to put in the record this afternoon, the further points that you Wc.nt to make, the 
debates you want to have with the purists, or the impurists. 
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THIRD PLENARY SESSION 



CHAIRMAN RISING: We are going to open the floor to anyone who wants to comruent 

about any of the group reports from this morning. I know that people were straining at 
the bit this morning, wanting to discuss things that were not part of their group report, 
and this afternoon I hope they are not too sleepy to remember what they were thinking of 
this morning, 

DR. LESTFR: Some years ago I spent an evening with the minister of health for the 

Kingdom of Nepal, aL.d during the course of the evening he Jascribed his several years 
of study of public health in this country ^ I asked him, of all the things he had 
learned, how he intended :o apply all of them Th^ sum total of his remarks was: "In 
my lifetime if I can achieve a single clean well in each village, I will have done 
more than I think time and capabilities will allow me," tn say nothing ol all the 
rest of the things he had learned about public health. 

In some ways I think that I, particularly, ir, putting on programs for general 
practitioners — or seeing that they are put on -- feel that, ves , we should measure in 
t.erms of behavior. I cannot see in the forseeable future how I can accomplish that 
for the State of Minnesota. Certainly some hospitals will have ways oF measuring this. 

One of the tasks I have, and I am sure certain others of you run into this, is an 
institution-preserving meeting, an annual meeting where head count is all important. 
Members must turn up. They must pay dues. They must meet one another for fraternal 
reasons. They must have a house of delegates meeting where they pass resolut ons . The 
instif.t ion achieves many other things for the doctors and we think, rightly or vrrongly, 
eventually for their patients. In chis setting we are asked to produce a meeting, and 
I do not have imagination enough to foresee how I could evaluate changes in behavior 
of physicians from all over the State of Minnesota. 

Like the minister of health from Nepal, if there some lesser method that 1 can use 
at this time to salvage some educational value from a meeting which I am not asked to 
evaluate. What can I do to i^^prove the educational value of their meeting? hTiat tech- 
nique could I apply? I would like to ask those here: is the pre- and post-test still 

valid with those kinds ot limitations? 

MR. JENKINS: I have ?. question. Are you really after an educational objective, 

or are you just out to get people together for a fraternal gathering? If this is the 
objective, a head count is probably a valid evaluation, if that is all you are out to do. 
If, indeed, you are out to change behavior, then the only way that you can know that 
you have changed behavior is to measure the behavior both before and after to find out 
if it has changed. You have to go back to the objective. Uhat is the reason for doing 
it? What do you think you want to accomplish? Once you have this down, it b’^comes a 
very clear task as to what you need to do. Tf your objectives are stated in what you 
want to accomplish, then you ju3t go about the task of measuring them. It nay be that 
the instruuents do not exist and you have to develop them, but I think they can, in all 
cases, be developed with some degree of expertise from people such as those present here 
today. If you want to have a fraternal get-together, that is fine. There is nothing 
wrong with th'^t. 

DR. EISELE: I object to that "either/or," either it Is a social gathering or an 

education experience which absolutely must le evaluated. You have another alternative 
there, an educational experience which will not be evaluated, 

o 
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MR. JENKINS: May I respond? Can you prove it was an education experience if you 

do not evaluate it? 

DR- EISELE: No. I do not believe I am always obliged to» either! 

MR, JENKINS: Then you cannot call it an educational experience. You are only 

guessing that it is an educational experience, 

DR. EISELE: I am not asking for a grant supptrc, so I am not obliged to prove it 

except to the satisfaction of my enrollees, 

JENKINS: I do not care whether you are or not. Is this an educational exper- 

ience here? I think it is. I have learned something. You are not going to measure It 
to find oute I think I could have by measuring some object ives» and we only had one, as 
I undetstand it, to write the report that is coming out of this. As a school teacher, 
sii , who was charged last year with evaluating youngsters, putting grades down, and you, 

I think, are involved ir. a medical college, are you not? You are going to give grades 
to students on what basis? 

DR. EISELE: No, We do not do that any more. 

ItRr JENKINS: You do not rank any students? 

DR. EISELE: No. 

MR. JENKINS: No ranking? 

DR, EISELE: No. 

CHAIRMAN RISING: May I ask whether you ure talking about evaluating the students 

or evaluating yourself now? Did you evaluate yourself as a school teacher last year? 

MR- JENKINS: I tried to, I was very inadequate at doing the job. 

CHAIRMAN RISING: Well, could you call it an educational experience? Is a high 

school education an educational experience or have we been kidding ourselves? 

MR, JENKINS: For the teacher or for the ....... 

CHAIRMAN RISING: For the children, 

MR. JENKINS: For the children? I don't know. 

CHAIRMAN RISING: I think this is Dr. Eiscle’s point. He may be taking exception 

to people saying: ’'continuing education in medicine Is not educational unless you do 

evaluate it," whereas for eons past they have not objectively evaluated any other edu- 
cational experiences — Socrates to graduate education in zoology. 

JENKINS: If we are going to make statements abou what is to be gained, in 

other words, why did we do it, if we are going f.o ask a docto ■ to come to it, we should 
have some reason for him /Tiling. If the only reason is that he will be exposed to 
someone's ide&s, then fine, a heed count is valid if t\is person did, indeed, present 
ideas. Someone is paying for that. That is the question that was asked this morning, 
and supposedly the physicians are asking, "who is paying for it?" I'Jho pays our salaries? 
In essence most of them come back to the taxpayer at some point, 

CHAIRMAN RISING: Some of them, inr.idencally , come from the taxpayer. That is 

msinly the RMP progr&ms. 

o 

pni^MR, JENKINS: No, all the people who work for universities 




chairman RISING: I take exception to that* The UnAv^ersi ty does not pay my salary. 
The doctors who enroll in cur courses pay many of cur salaries. Dr. Eisele^s included, I 
feel surer 



MR. JENKINS: All 

CHAIRMAN RISING: 

MR. JENKINS ; How 
to Medicate? 



the doctors pay those salaries/ 

That is right . 

much of the doctor's salary comes as 



a result 



of 



taxpayers ' 



money 



DR. PERhMAN: I would take exception to that. How does the doctor pay your salary 

and not the taxpayer? 



CHAIRMAN RISING: Dr, Eisele's prograiii, the Minnesota program, ruy program, the 
program at Albany, a good many of the large, maybe not educationally successful, but 
quite popular programs, are paid for by fee income. Doctors pay fees, 



MR. JENKINS: O.K. I d“fsagree, because most of the fees ate not paid by individual 
doctors, they are paid by... it Is not tax-supported, it is taxpayer-supported, citizen- 
supported, because it comes from the hospital till. 



CHAIRMAN RISING: Now wait a minute, Money for cur programs does not come from the 

hospital till. 

DR. CLEtfENT BROWN: To follow up on Keith's point, l^iere is the money coming from? 

Suppose it does come from the physician; suppose it comep from time out of his practice, 
rway from his family. I think one could make a good case that we are really perpetrating 
some gigantic fraud on physicians, getting them f.o spend time away from their families, 
away from their practice and from the patient care they could be delivering to attend 
courses that we have no idaa whatsoever whether this has any influence on their be- 
havior, on their patient care, on anything, if we do not measure j.t, 

I think we must be concerned. We may oe per| eurating a gigantic fraud on the 
American public, the American doctor, and everyone concerned. J am not saying that we 
are; 1 am saying that this may well be the case. As a matter of fact, almost every 
time an educational experience is evaluated a reasonable kind of way, is not this ex- 
actly what we find? Ihe New York television programs, the Butterworth thing, the 
McGui re-Babbltt thing that they evaluated on cardiac auscultation, e^^ery time we measure 
somethiig in a reasonable kind of way regarding continuing education we find it has no 
effect at all. 



CHAIRMAN RISING: Those things, Incidentally, were net financed by the enrollees. 

1 would like to say one thing now that was said in our group yesterday. 1 have not 
heard it repeated here, 1 believe. That is, that the "evaluators" here probably go to 
their doctors and trust them to make what may be a life or death decision about him, but 
he does not trust the doctor to have a fair idea of whether a program i^ educational to 
him. 1 submit that, if a doctor spends his time and money going to an "educational" 
program, there is reasonable expectation that it is actually educational, and in that 
sense a head count has at least slight validity. Nobody would say that if vou can 
economically evaluate the change in behavior, you should not. Probably anyone would 
agree to that. The "either/or" positi'^n and saying that we are "perpetrating gigantic 
frauds" are extreme and unfair. 



The programs put on in your hospital, Clem, are an entirely different thing. There 
Is no questiOTi that you are 100 percent right in a community hospital, where al^ of us 
wi.Tl probably agree le where most continuing education should be. I have noticed here 
in this group that, in spite of having pretty good luck in getting a mix, we have had 
a woeful shortage of coaanunity hospital medical rducatore^ and nonphysician health 
educators, We have had very little contribution about them, and all the time we have 
* ' Q try to transpose "physician" or "doctor" into "health worker" or "health profes- 
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HR. HERZOG: I think the phrase ’'perpet c ac lun of a fraud” is a little strong. It 

makes a point, but I think the point is that you are raying that you have got to some- 
how justify the expenditure If you can not justify the expenditures on continuing 
education how do you ki ow where you are going: It is my viewpoint, unless somebody can 

change Ic, that I am not the least bit more confident that the money spent on evaluation 
has been well spent because I have never seen the justification. I have not seen any 
more good evaluatiors of evaluations (ones that really measure what they set out to 
measure) ih in I have for continuing education. I will take the devil's advocate role 
and say you have both got to prove thzs to me because 1 do not see the value. 

DR. THOMaE BROIv^: I wonder whether anything that happens to anyone lacks educational 

content. 1 think that all of our experiences are a lybetnated kind of thing that, you 
know, we feed out and we get back. The question that we are addressing ourselves to at 
this conference is: "what Is the role of evaluation in the e>ducational process?” VTnat 

can we make observable, and what ought to be made observable so that we can use the data 
from evaluation to improve our programs, that evaluation has a purpose, I think any 
experience is educational. It may be negat;ively educational, it may be positively educa>- 
tional, but it is educational. I think we have to take a look at, "vhat is the role of 
evaluation, the data that comes out of evaluation that we car feed back into our programs 
that can make us more effective.” I do nor think anyone really wants to disagree with 
that . 



CHAPtMAN RISING: Incidentally, it may be related to how you can better intrigue 
professionals, health prof esslocrals , with ycur program, how you can get them involved. 
You can have a marvelous teaching instrument, but if no one uses it it is valueless. 

In the community hospital where there is a certain amount of clout, you can make people 
use it and they will begin to appreciate it. 

DR, VANSELOW: I want to support what Bill Her^og said- Clem, you know it is nice 
CO get up and make statements that none of these things hax^e any educational value,.. 

DR, CLEMENT BROWN: I did not say that I said, maybe we are perpetrating a gigan- 

tic fraud, I do not know* 



DR. VANSELOW: The thd ng that concerns me is out measurement devices Are our 
measurement instruments accurate enough really to measure what we think we are 
measuring? That is one thing that really bothers me. 

The other thing I would like to do is get back to Jjhn Lester's comment. John, I 
really do not think we can expect you in Minnesota witli the Academy oi General Practice 
to do the type of evaluation we have been talking about here, noi do I think we can ex- 
pect the great majority of people who ate djing continuing medical education to do it 
fo; every program. 1 would like to see some studies done from which we could generalize 
so that we can say that, if a program is put on under the following cl r cams tarces , it does 
or It does not have any Impact. It seems to me that if y.^u could do that, if you could 
put the money into a few studies and then generalize fjom them, you would be In better 
shape = I cannot imagine, however, how we are ever going to be able to do tlie meaningful 
type of evaluation on all of our activities. 



I wonder, I think Hugh Petersen suggested this this morning, whether this Is not 
something that this group ought to recommend: tliat a cooperative study he set up where 
these various techniques can be evaluated with the hope that w^e can generalize and not 
have to evaluate everytiilng from there on. 



DR. LEMON: It has been curious to me today and yesterday, as we have been talking 

about evaluation, that we have frequently been moaning that there are no good studies 
to hark back lo, but no one up to this point has mentioned the fact that at least a 
fairly well sketched out study r^as done right here in your bailiwick entitled "Die 

— iology of Continuing Education” by Chuck Lewis. LT\en I first teac. it — that was 
the days 1 spoke about this morning when I thought I had better look for a place 



r-i 

/o 



to practice After twenty-five years, wj-jld you Like to make a fe-' vs^eJl chosen comments, 
in four-letter words, ab'i’t this thing which demonstrated that you had not made any im- 
pact in Kansas for twenty-five yaacs^ 

CHAIRMAN RISPIG: That study was not done behind my back or without my knowledge, 

and it was not published without my 0 K. 1 read it in manisoc ipt and t argued with a 
few of the statistics, because he used the data differently, I thought, than they prob- 
ably should have been used. This was not a question of measuring behavior and end 
results, I think Chuck would be the first to ceil you that Kansas w s one of the few 
places in the world that he could have get the data to d^ such a study. becr>use we had 
been doing enough evaluation of at least a "matkec rvaluat ion” type -- and tais is the 

term he used — to make this mucti possible. It would be diffic.u:t fci me to prove to Dr r 

Petersen that we had changed behavior of doctors in Kansas. I it vjouid be dif- 

ficult eveu it we had had some pretty sha.rp instruments because the field is broad, both 
geographically and in the numoer of persons invoiven, and there ate coo many contaminants. 

1 have said, and I would like to say again -- 1 have said it on ihe same platform 
with Clem Brown at Wes Eisele*s place at Estes Park when he was having die program — 

that continuing medical education is not a course or a set of covraes. Continuing medi- 

cal education is something that goes on continuously and in many different ways in each 
instance^ It involves, as Don Williams, who headed c«_ntinuing medisal eduiation in 
British Columbia many years so aptly said, work In the doctor's own study, ^ n his 
meditation, his reading, it involves consultations with colleagues It involves curb- 
stone consultations as well as forma) consultations- In many, mau ways learning goes 
on. For continuing education to be truly effective it needs to go on at the home base 
of the doctor or health worker, which means generally in his own community and in his 
own hospital. It needs to be an evr cyday thing, not a course thing. 

I would like to repeat an aialogy with religion that is old I see the lole of 

most ’’courses" — and this includes the courses tfiat the organizations And institutions 
put on and the voluntary health agencies put cn -- as analogous to revival meetings. 

People go to 3 revival meeting tor various reasons They go Lj feel uplifted They go 

to courses because of intellectua) Uf>Ilfc This :s not necessarily directly educational 
but it might indirectly be because s as in the case religion, it may influence what 

^oes on when they get back home, wVien they g: back to their own cominunlLy, their own 

! spital, their own home 

If we Inspire people to study a little more, to have more c onsul tat ions , there is 
no way that we can, within any reasonable cost, evaluate ■ hange of behavior because 

we may be talking about the use of diuretics, but it may inspire *^liem to get involved 

in something else about hypertension and into stroke and into t ehah i 1 i t at ioa by a cir- 

cuitous route. How do you evaluate that 

This "either/or” business dees bother me, and to think that all C/nrinuing education 
occurs in courses or in relation to RMP programs ur to medical scliuc'i prograius is ob- 
viously fallacious There was continuing cdu».dtlon before there were schools. It has 
gone on since at least Hippocrates and probably before that. Uliat they r aught might not 
have been very good, but I am ‘ ot sui ? that what we are teaching now is anv good. Five 
years ago it was great, but we know today that most of it was pretty lousy If we had 
tested and saw we had changed behavior, we might have seen tiiat we were changing behavior 
In the wrong direction. This has been mentioned many tlm^s. What a*'e your criteria? 

Five years ago cycTamates were o,k. and tolbutamide was great, but now they seem to be 
out ! 



PR. PERLMAN: I a.n interested in Svmetiilng that one of the groups brought out this 

moaning, but we have not explored. There ate tools available, t. thin*, you just said 
that, but we cannot neccessarlly make sure that the tools ate used. We have to figure 
out how to get the average community hospital and the average pliysician to be knowledge- 
these tools and to use them. 




I would Jike to ask the group to explore what role the university, the medical 
schools and postgraduate education -- not just the postgraduate education groups that 
are represented here, but the medical schools themselves -- have in teaching, and how 
they can gc about teaching physicians and medical schools to use the tools, such as 
medical audit and self-evaluation. 

DR. KORST: This is a very important point. It seems to me t’^-'at continuing educa- 

tior* is really a frame of mind, and this is something that needs to be instilled in the 
student at the time he is a student to continue on in practice. One of the most import- 
ant things X Ink we have talked about today io not only getting to the physician in 
the community hospital — and I say this because I am a director of an education program 
in a community hospital but I am associated closely with the university -- but in the 
medical school we do not do this, we do not teach students about how to continue their 
education cr the importance of it. We do not show them ways to evaxuate, although stu- 
dents are beginning to get more into self-assessment programs. The students now tell 
us that they want the pass-fail system but they want examinations. They want to eval- 
uate themselves once or twice a year, have us show them, with them, where their weak- 
nesses and their strengths lie. They do not care about the grading of this, but they 
are very anxious to have a seli-assessment , 

About auditing, I think this Is just an addition. It would be all right to talk to 
medical students about the medical audit in terms — maybe Dr. Brown will take issue 
with this — but I think a great deal more has to be done before we can say to medical 
students, ”now this, in effect, is something you are going to be working with.” This 
has to be proven. My own feeling is that it would be very acceptable and very good, but 
it has not struck me that this is generally accepted by the medical profession as an 
ample way of evaluation. I think it should be looked at but I do not accept it with- 
out reservations. 

CHAIRMAN RISING: It is rot really accepted by manv coirinunity hospitals! 

MRS, SHORT: I have two points to address myself to^ one being that I am partici- 

pating as a representative of Vanderbilt at this time; but prior to my association with 
Vanderbilt I was associated with a community-based educational program that bticame firm- 
ly established and grew through the evaluation of the doctors who attended, who re- 
peated their attendance, and who verbally expressed themselves to the effect that they 
came back because what they found was quality and practical application In iheir 
practices at home. 

Whether it is done formally or not, evaluation goes on all the time by the attendee 
and by the program producers, and because oi the increasing social problem and r.he in- 
creasing social interaction that is malting this more of a public problem, we are going 
to have to have more concrete tools, but you cannot get away from the fact that evalu- 
ation exists. We are not putting it to the best use, however, and to ti',e most practical 
and useful methodology where we can more concisely employ it. 

My other point that I was interested in expressing was that it seems to me that 
a let of the comment has been directed to evaluation as something that occurs far along 
in the process, kind of after-the-fact rather than before-the-fact, and to me evaluation 
is part of th^ total process whereby you may establish a well-defined, concise goal, and 
as soon as you have that goal established the intermediate steps faVl into place and still 
leave you leeway, but the evaluation is built into the program; it is not tacked on at 
the end . 



DR, FIFER: I would like to say if we use only behavior as the means of measuring 

success of our efforts, we could be misled a little. 1 talked with Jolin Lester about 
this earlier. 
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Thi^ is my knowledge of> say, infectious mononucleosis that 1 know something about 
right now* If I do notk.-.ng about my own maintenance in infectious mono my fall-off 
cuive is like that (see graph). If you measure the change in my behavior, at the last 
point my behavior will have changed negatively. 



X -X X X— X 




X = knowledge maintenance 
o = knowledge fall off 



Years 



If I go to enough courses on infectious mono where I perform what might be called 
"knowledge maintenance/’ assuming that there is nothing new, then my curve is going to 
be like this, and I am going to be taking care of patients vitl\ Infectious mono five 
years later at the same level as before. If you measure my behavior it will not have 
clanged. If 1 did not go to those courses and you measured my behavior you would find 
it deteriorating, I do not know how yo\ are going to take care of what occurs in know- 
ledge maintenance, and there must be quite a bit of it that occurs. 

Another thing I think about behavior is that we are about, because of the pressures 
of the moment, to give a course in the use of h-dopa in Parkinsonism, a disease which 
is a terribly important thing for practicing doctors. Because L-dopa has suddenly be- 
come available it ic an important thing, it is a one-shot, and if we are at point A 
in 1970, and you do a behavioral analysis of what doctors do with Parkinson's disease 
today, you will fird that X many use tnis new agent, and a year from now you will find 
this new agent being used by Y many doctors, whether we have taught them anytning or 
noc. They are going to use L-dopa, not because of a course I give. There are other 
places for them to get good advice, but they want our help and our reinforcement of 
what they get from other sources. Ine doctors are going to be using the drug whether 
w€ give the course or not. We would like very much to have them apply it well. 

y! b 



1970 19?1 

This is an example of two behavioral measurements, neither of which may have any- 
thing to CO with the course I have interjected; in fact, one of which could be a re- 
verse correlation. 

OR. PERLMAN: Two comments, now that Bill Fifer has spoken. There Is a definite 

measurement, par ticui:ivly in your first example, Bill, because the important thing in 
evaluating, self-evaluation and medical auditing is to mesh against the patterns of 
cfire that you expect. If you expect that the idea] care for hepatitis has not changed, 
that is what you expect; therefore, you have measured the behavior, Merely maklrg the 
assumption that knowledge will drop off if you do no*" do anything, if you do not use 
aomethi.ig, a relationship implication can be established that we have done something. 

It will not. be negative. 

I really wanted to talk because I am very conco'^.ied about the thing that Dr. Korst 
said. If I understood him correctly, I find myself falling into two possible conclusions, 
and they sc<.re me. The first is that we ouglit to be very careful of these immature 
medical studentc because they cannot evaluate. Urtf.l we have proven that self-evaluation 
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and medl-:al auditing is any good, we better not ey.pose them. I am probably being a 
little strong, Don, but you could get that implication out ot what you had to say, and 
that does bother me. 

From a personal standpoint, because of what I dOj I chink : has been pretty well 
proven in many hospitals, and Clem Brow*n certainly is an example, that medical audit 
is an effective tool in hospital evaluation. The first thing that concern^ me even 
more, whether it has been proven or not to anyone's satisfaction, is the fact that it 
does work. 1 think we have to expose our students to the possibilities of something 
that is not jusL blue sky; it is something that has worked in some places, that cer- 
tainly can be improved, as anything can be improved upon, and teacii them this concept* 

I do not know if you were trying to stimulate or what because you are the one who 
brought up self-evaluation of medical schc-Oi originally . 

DR. KORST : I was hoping 1 would get some lommenis like that. That is why T said 

it. 1 really said, or what I meant to say, is that I an not opposed to the 

students being taught this concept^ 1 think the students ought to look at this as a 
research in health care, just like we are teaching students to work with computer inter- 
views. I think the students would take hold of this and work with it very well, I do 
not think we should present this to the student as a matter of fact 'h_his is how we 
are going to evaluate." We have to investigate and study it. I did not mean that the 
students ^houjd not have it. I think they should have it but in a way in wliich wo are 
looking at a lot of these programs. 

DR- PERLMAN: One quick answer to that* 1 do not see it as being an "e i ther /or . " 

1 do not think it is the same as computer interview, because »_orL";puter interview is not 
available at this time fiom a practical sense in any community hospital it it is not 
their own experiment. That is true with medical audit. Medical audit m some form 
other is probably available in f^’fty percent of the hospitals in the United States 
The tool is there. It may tiOt be used well, but the Loul or Mn mech.anism can do 
something It is there, and it is somewhere in the middle of the two. 

DR, PETERSEN: let us not misunderstand the use of the behav^ioral measure. In 

using it. we do show a difference, and we do know what the difference means, that the 
physician is behaving differently. That was the point. Now, we can specuiace about 
WHY, and of coutse we have a controlled study which he has used, noa-exper ien. ed or non- 
continuing experience with continuing experience, operation defined as "continuing edu- 
cation courses," ot attending these courses and not attending the courses. It just in- 
jects a note of certainty into what ve are doing, That is ill 

DR. LEMON: A couple of paints about behavior whivh 1 think were stimulated by 
this, and 1 had hoped that this might come out Df my question about Chuck Lewis' paper, 
because it w^ould seem to me in reading it, that since we were measuring in fact an 
end result, I have forgotten the details, but it was deemed that the impact was "non- 
successful," because no change had been observed. This po.ints out the tact that main- 
taining the status quo mt^y in fact be an objective that is worth reaching. These indi- 
viduals who did njt Improve in Kansas practice also, perhaps, did not deter loraie over 
this period of tire as a result of their exposure. That might be w’or th something. 

The other thing 1 think about behavior, at least the effect on patient care, 
which i personally think is the only reason for being in this business, nevertiieJess 
the measurement of it doe.s not have an invariable relationship to knowledge. 1 think 
we ehould keep that In the back of out minds also, because there are several reasons 
why people do not behave to a certain standard, and they are no: all due to the fact 
that they do not have the knov?ledqe. That has been said In several different ways, but 
it does not hurt to emphasize the fact again. 

We have a delightful person in my part of the country vlio cells the story about 
talking to a farmer, who, in turn, had talked to one cf those agents who drive around 
do ycu call then? -- a farm erttension agent- He had made the eficrt to get a 
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farmer to Improve his farming practice in a particular area, and on several occasions 
he got into discussions on specifics with him. About the fourth trip around the fellow 
saldi "Listen here, young man, I already know how to farm ten time^'' better than I am 
doing." So, he knew better how to do the farming but he wa^^ n^t doing it. 

CHAIRJIAN RISING: I an. sure yoj wanted a little more comment, and I would not ar- 

gue about the conclusions that Chuck Lewis came to in that paper except to say that he 
was ].ooking at a rather narrow area, and was not evaluating the type of thing that I 
would like to evaluate. He was evaluating from a point a view of a public health per- 
son. It Is just conceivable, and we have mentioned severa) frauds that have been per- 
petrated, It is just conceivable that if you evaluate an educational program and find 
that learning has not occurred or that behavior has not been changed, it nay ii/iicate 
there was something wrong with the evaluation. Perhaps you were looking at the wrong 
things or using the wrong criteria. Evaluations probably are not always perfect, 

DR- BARSON: 1 am getting apprehensive that somehow the proceedings of this 

conference report will be something like: "These are the techniques which should be 

used in evaluating medical education; therefore, if you are not using them you are 
really in the wrong ball park." I glean rrcTn a number of comments th.''t there is more 
to evaluation than the instruments possibly have to offer, that eventually, like the 
term "the unexamlned life is not worth living," basically evaluation ability is a 
personal attribute which you try to stimulate or enhance in a person, and it should 
not be confused nth the measurement techniques or devices or the indices which we have 
established , 

I say that in light of the fact that we are not even certain winch evaluation 
Instruments to use, how often, or vrhen. Typically, we make it a terminal activity, 
whpi'i actually we all know it is a continuous activity. Perhaps we should not of er 
an examinatnn at the end of a course any more chan you offer the measurement of a 
person’s life worth after if.s fifth year. I mean, it has not lived its entire terra 
cut, It nay require a st^quence of experiences and, therefore, the measurement at any 
one point may really be irrelevant tr the person's potential. 

I am worried about that because 1 think, if anything, I would like to see In the 
proceedings that develop Iiere tlilc essence that evaluation is a response to an experi- 
ence, both to the individual and to ttie observer, as opposed to the pui ci ■’‘.ns trument ali- 
ties that we have to brl ig. 

] think the funding agencies that back up educational pregrams are tending to be 
misled along this line and, therefore, are satisfied with a minimal expre.ision of evalu- 
ation. I would call for more data on evaluation — not Just more instruments listed in 
the appendices somewhere — but largely w’hat sort of behavior can you expect from the 
participants, botii the staff and the learners, I am very worried about this as I 
see ourseb "’s constantly battling over the instruii'entalitl es 

DR. BANK: Several times medical schools have entered the discussion, and my own 

particular point of view is to support Jess and Lis slogan ot "Forty Years, Not Four," 
My point here, and 1 think, Jess, you are the one who at least coined that phrase, if 
we are talking about continuing education we are talking, by definition, about something 
that is continuous. It does not stop at graduation from meaJ.cal school or from any 
school. It is part and parcel of a profession. 

I take, as a definition of a profession, one tliat sets standards that are higher 
than those v»ho receive tlie services of the profession have a right to expect, which 
means, if you are a physician, it is your obligation to continue to provide the beft 
possil le service, In this day and age vhe.i Mr, Oppenheimer canie out with a gra :d and 
glorious figure that the half-life of information is about eight years -- which we kntv 
in the sciences is even shorter than that -- there is a problem. Change comes about 
It has to be a continuous process. Education has to be a cont.nuous process, and it 
^ be a responsibility of educational institutions to provide for that continuing 
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There is one medical school, which will go nameless as far as I am concerned, in 
which I have talked with their people and they say: ''We are not Interested In continuing 

education one bit. We see our role a} working in research and teaching medical students, 
and when they get out of here, that is itl We are not concerned with them further." I 
cannot subscribe to that point of view of an individual physician or any individual who 
says: "Well, I have now been graduated. That is it. I know everything tht^re is to know." 
That attitude cannot exist in this modern society. 

To get back to my original point, I do not know how this philosophy can be ingrained 
in medical school teaching. The point was made about PAS — whether or not PAS should 
be put forth as a technique may be questioned — but can you argue against having the 
principle of something, however imperfect that measuring stick might be, to use as a 
guide to go on and have some basis on which to try to improve yourself? 

One other thing, while I an\ on somewliat of a soapbox: can we, through whatever 

evaluating techniques, say that ue have in fact improved patient care? 

During an earlier discussion a statistic was used; "Well, you know, we cannot say 
that we have increased the longevity of American citizens through all of the millions 
of dollars that have been spent in continuing education." This is the wrong statistic 
to use. If we wanted to use that as a measure, let's do away with automobiles. There's 
a way of increasing longevity. Someone used pneumonia as an example. You can have the 
best possible care for pneumonia, and if that is the thing you are looking at you should 
only look at that alone, riot all of the other things that can influence patient care. 

One study i am sure all of you are familiar with, one that was done by Katz relating 
to the introduction of a particular drug, it was a highly controlled thing, and, the 
point being, this new drug, an antibiotic, was not being used in the community. They 
had done everything, they thought, to get this drug introduced into the community. 

All of a sudden they found a sudden upsurge of prescriptions that were written for the 
drug. The reason, they found, they were suddenly getting this increase of prescriptions 
was the fact there wa:> one physician in town who, Ui^beknownst to the researchers, was 
the one that everyone else looked up to, and the minute he wrote the first prescription 
for the drug, everyone else followed. 

These are among the contaminants to which you referred, Jesse. There are millions 
of them don't hold me literally to that number — but there are many, many of these 
kinds of contaminants that you never know. 

Probably the g.:eatest definitive study on the flow of information is in the story 
of hybrid corn. There is probably no open-poll f rated corn grown to any extent in the 
Unirad States. It is ail hybrid corn. There are about seven steps in this whole 
process that have been identif-^ad. 

Thera are those who are early adrpters, there are those who are late adopters, and 
there are some who hardly ever do anything at all. 

DR. ROYER; 1 think this Is an appropriate time to present a thought that I have 
developed over the last eighteen hours. I would like to speak on the topic of physician 
attitudes toward evaluation of continuing education, 1 briefly raised tills question 
yesterday in our group discussion, and seat it up as a trial balloon, but the :^mportance 
of this hos grown on me during the last eighteen h'^urs and I would like to pucaae it. 

Several persons have spoken of phyf Ician resistance, and I gather t.here is consen- 
sus that such resistance does exist. However, 1 rubmit ^his resistance to evaluation; 
the resistance to continuing education factor is more complax. More deep-seated than 
we have described- 
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Specifically, I question the interpretation, and perhaps the \riJidity as veil, of 
Keal Vanselow's second assumption in his paper; namely, the assumption tnat the practi- 
tioner is genuinely interested in the quality of medical care he delivers. I suggest 
this falls in the '^motherhood-flag" category, and any physician facing a list of top 
ten words, as was presented in this physician study in Michigan, would be amiss not to 
include those items related Co quality medical care, "Yes," the Doctor says, "I am 
certainly concerned about how good a job I am doing, I certainly am concerned about 
keeping up to date.'' Accordingly, t:r3u, I question the interpretation of these data. 

The questionnaire does not indicate what priority these concerns enjoy in the physician's 
life and in his practice. Herein I find a real discrepancy. I submit that among his 
priorities, concerns other than upgrading quality care frequently overshadow’ desire to 
keep up-to-date medically. Some of these have already been alluded to: golf, stocks, 

politics, etc, I think the discussion topics that you witness in doctors' lounges in 
community hospitals vrould surely give another indication of peripheral concerns, con- 
cerns other than health care, 

I suggest, then, there are more complex reasons, some of which have come up in our 
discussions (namely, icasons related to a political-social conservatism:), which resists 
change, and unless we adr’t that such concerns — these deeper ones, more complex ones 
— often take priority over that of upgrading health care, we are at a loss to explain 
the gap between the questionnaire results and actual medical practice. 

How’, a couple of examples, we find the physician paying lip service to on-going 
education who continues to use chloramphenicol or aminopyr ine , and with the explanation: 
"I have used these drugs for the last five years and I have not gotten into any 
I haven't seen any bone narrow depression," 

Another example, we f:.nd the physician subscribing ''yes" to improved health care, 
yet very easily writing off all alternate systems of health c^ire delivery -- Kaiser 
Permanence, Medicaid, and closed panel practice. 

So, I underline the gap between the goals stated and the actual performance, and 
this is the same gap that we see between test performance and actual practice, A dis- 
crepancy betw^een management of simulated patients, for instance, this came up cui 
group discussion, simulated patients, whether it is mec!anical patients or whether it 
is a paper* pencil simulated problem, and the actual practice of that physician. 

Example: The treatment of a sore throat. On paper, of course, we would all get 

a cuJ.ture, In practice how many times do we actually call on the phone, "0 K, 250 mg. 
q.i.d,," without even looking at the sore throat? 

Regarding physician attitude, I hear a note of pessimism, and unfortunately, or 
perhaps fortunately for patient care, the sense of urgency that we have talked about Icf 
echoed in the various me 'la. 

In packing, 1 threw in several things, 1 was reading all my spare timie these fev; 
days, one of which I was delighted to find has three or four articles on health erre. 

It is the Saturday Review for the week of August 2A. I assume from the nodding that 
some of you have read these* Senator Ribicoff talks about the Infant mortality and hovr 
our health care does not compare to other programs. Dr. Knowles then addresses 'limself 
to, well, it is really a broadside against organized medicine. Carl Cobb, writer for 
The Globe , talks about the shortage of doctors and what you find and makes a plea for 
compulsory insurance. 

I would like to mention a few things that bear directly on this point that I pre- 
sent. First of all, Senator Rihfcoff says: "All that keeps the medical care system 

afloat is the fact that millions literally have no li.iowledge of their medical needs" -- 
Our present system. 
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In the Knowles article, what he finds cbj ectionable in organized medicine is what 
I am trying to describe for the individual in practice, too many individuals in practice- 
Several quotes, please: "but v:e are conservative, we doctors, and we do not want change 

and we do not want to face certain facts," 



The broadside at AMA: "The ANA has resisted every major social change in medicine 

over the past fifty years, since, incidentally, 1916 when the AKA was on record favoring 
compulsory health insurance." 

If organized medicine continues to maintain its posture of negative vigilance he 
fears destructive confrontations of an angry public and ultimately the complete social- 
ization of medicine in America, He says that, however, much as the physiciai wants to 
preserve his autonomy and self-determination, he will be forced to surrender some of 
his authority and work with others. 



\<hat I am saying here is that the "sell" w^e have been talking about, this desire to 
sell evaluation of continuing education to physicians, I find myself quite pessimistic 
about. Perhaps we are naive in that we think we can sell this type of approach. I come 
back to some of the comments that have been made about a broadside approach where we try 
to remodel the whole system, perhaps priority being at the medical school level where w^e 
work for this attribute of on-going education that has been mentioned by several of the 
speakers . 

CHAIRMAN RISING: I would question whether we have been talking largely about how 

to sell it to physicians or how to sell it to educators- I would judge that the edu- 
cational establishment is at least as cupable as the medical establishment which, in- 
cidentally, is lu t really represented here very heavily, 

DR, FIFER: I do not want to refute a lot of points but, unless there is someone 

here from Chicago who can successfully contradict me, I believe that Cliristine McGuire 
has got data from practicing physicians which indicate that a PMT, a patient-management 
type of simulation on paper, does actually bear a very high correlation to what phy- 
sicians do in practice. I believe thi/. was done among practicing physicians, I think, 
in Rockford, Illinois, If anyone knows the exact reference I would be glad to hear it. 
If we did not Imve to keep me.isuring behavior every time it w’ould be fine, if could 
go back to a patient-management simulatoi on paper, it would be fine, but I believe 
that Christine McGuire has this information, not only about how: medical students and 
house . "ficers behave, but how practicing physicians belave, I w^ould like to hear some 
evic . - that this gap exists between what we say ire do, a la throat culture, and we do. 
If there is evidence that there is a gap, I would like to hear it. 

DR. PETERSKN: The general izabil i ty of your results is directly dependent upon 

your sampling techniquer, your sampling procedure?, i.nd if you do r t have random 
sampling involved, then you cannot generalize your results and it would not na' ter \chat 
Chiistlne McGuire says. 



MR. COOLE: Dr. Fifer, it is unfortunate that these studies are based on a very 

litn. ted sample and are not, in fact, generalizable , They indicate a trend toward a high 
degree of validity between actual practice and patient-management problems, but as of 
yet there is no generalizable study available in this area. 

DK. TERKLAN: Our organization has done probably twenty or thirty studies which 

will generally point out this trend, Tb.ere was one that I was involved in that I 
think was a good example. 



We did a study on use of antibiotics in patients coming in for routine surgery; 
namely, hernia, appendectomy, hysterectomy, cholecystectoi-iy , and hemorrhoidectomy. Of 
course we did not do a survey to ascertain what physicians lelt they .should le doing 
or thought was the right practice, but according to the leacned scientific articles 
from all the medical schools you use anti-inf ectives infrequently. 
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Across the country, of half a million patients, so we do not have to worry quite 
as much about sampling validity — I am trying to remember some of the figures, and this 
is> teaching end non-tenching hospitals, very little difference — nearly .?0 percent of 
appendectomy patients; A8 percent of cholecystectomy patients, and all this type of 
thing. I think the gap that Dr. Royer is talking aboiu dcas exist in manv areas. If 
this information can be challenged it is on the basis of under-recording, so it 
might even be worse than the records show. 

DR, IRWIN BROWN: For a day and one-half I , as a paranoid physician have heard 

my attitude and ny moral finer and everything attacked, but the subject iis Evaluation 
of Continuing Medical Education. We can focus on the physi-'i^n because bo. is vulnerable. 
I am serious about this. Me can talk ibout hir.. We can say that more kids die in the 
first year in the United States than they do in Denmark, or wherever it is, but let us 
be sure we are applying the same uieasurements . Let us say we are counting all of them 
that were hatched out as being dead, or whatever, and count crat into oui statistics and 
not wait until ten d s later and start counting the survivors and apply it to another 
kind of thing. We have got to talk about the same thing. 

CHAIR>[AN RISING: Excuse me, are you saying that they measure inf.uit mot tail ty dif-^ 

fetently in Scandinavia than they do here? 

DR. IRWIN BKOk^: Yes, they do, and then we come up and say we a:. e terrible, our 

bah ' >s die and the others do not. There is a difference in how you count th If a 

kid is dead the ninth day he does not go into some countries' vital statisti.es, and if 
he is born dead here he goes into our neonatal deaths. 

We are sitting around trying to see how bad thase doctors are, aid nene of them 
is motivated well. Of course a doctor is getting paid for taking care of sick people 
and his motivation in Fome cases Is money, but I like to think in a lot of rases it is 
patiant interest He is concerned; at least tha» is how he got into *t. The educator 
seemi; to think that he, the educator, is the only one who is weM noti\ated. Every one 
of us is hare because he is getting paid to do this, so it is money that ds doing it 
to us, too. Let^s get that straight! 

Now, how are we going to worry about evaluation in continuing medical education, 
which is what we are trying to do, if we are Just going to measure the doctor. I have 
not heard anyone say, "let us have the phy Iciauis as a group get a committee together to 
evaluate eoucators and see how they are doing, and study their motivation and their 
monetary interest, their Interest in getting grant money. 1 think we are all people. 
Doctors are not gods that are fallen; they are Just a bunch of poor people walking 
around trying to make a living like you are, really, I think we really need to get in- 
to not Just talking cbout one segment of the health care field and his motivation. 

What is your motivation? 

MR. JENKINS: I am an educator, not yet in medical education, although it looks 

like an interesting field. 

I am in sympathy with what the doctor is saying. I s« id this yeatetday, and 1 
think it e.hould be in some of the iiotea that were presented this morning, at least in 
the notes that were taken yesterday, that we are hung up on one thing, and that is 
hanging the doctor up here and taking a shot at him, and you know, this makes a good 
whipping boy, particularly when there are 'ot many vf them here to argue back. Unfortu- 
nately, the only way ve can measure the effectiveness of the educational progran is to 
see, Indeed, what happened to him, because he is the recipient of the activity. 

If I may use an example, and so that we take it out of everyone's harig-up, 1 will 
use your children, since I was a high school teacher. kTiat happens to your children 
when a teacher does certain things to them in the classroom? They go In, they teach, 
and the children a^e supposed to learn, and they evaluate, and they assign a mark of 
some sort — pass, fall. A, B, C, D, F, whatever you want to use. To tell us how ef- 
fective the teacher Is, how effective the educational program is, we must look at what 
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has happsned to the recipient or the student. Yes, I can say, "this teacher is good 
because he presented this fact and this fact and this fact, and he used these visual 
aids, and he did these things, he went on these field t *ps , and all of these activities.” 
if that child's behavior is not changed, he did not do a damn thing I He wasted your 
money. He wasted my money. 

Gentlemen, I have to come clear back to the original argument. I do not care 
where the money comes from to do the job we are doing. I do not care whether it comes 
from fees that doctors pay because they are convinced, you know, we are nice guys, lo 
they are going to give us money, or whether it comes from taxpayers' money. It's tioney. 
It's resources. It's part of our gross national product that everyone is concerned about 
right now, and it Is causing inflation and the whole rest of the thiiig. Ue have to be 
concerned about "do we get our money's worth?" 

I am not worried — well, yes, I am — about being able to prove that we did what 
we want, because if it cannot be proven I say "wipe it outll" If they cannot prove I 
am doing my job, wipe me out because it is your dollars that are paying my salary. 

CHAIRMAN RISING: Excuse me, did I hear what you said, that t^icy cannot prove you 
are doing your job, or that you cannot prove you are doing it? 

131. JENKINS: If it cannot be prove.... 

CliAIRMAN RISING: If you cannot prove it? 

MR. JENKINS; If ^ cannot prove that I am doing the job I am supposed... 

CHAIRMAN RISING: You changed "they" to "you," or rather "I." 

MR. JENKINS: O.K. If cannot, within the situation, prove that we are being 

effective, then wipe it out. It Is not worth it. It is a farce and it is a fraud, 
to use Clem Brown's term, and that, as far as I am concerned, is a valid term and I 
will accept it. I am using it, too. 

DR. CLEMENT BROWN: I said "maybe." I really do not know. 

MR. JENKINS: If thesa things are true, every time it is measured ve find out it 

is, and if these things are true then it is a fraud, because someone is being "cenned." 

Let us look at it. I don't care, you know, we are all worried about, "can we convince 
others, and can we convince ourselves that we are valuable?" Yes, we are valuable as 
human beings. We just may not be doing the job right. What is wrong with that? We 
fail. V’e cannot all be winners, but we can all try to do the job right. WTiat I hear 
in here is some people protecting, some people attacking, and nobody being willing to 
sit down and say, "O.K., now ie there a problem?" First of all, let us determine, "Is 
there one?" 1 think I hear there is one but maybe I am biased, i am an attacker, I 
know that. 

All right, let us first of all determine, "is there a problem?" If so, how can 
we correct it and what, then, becomes our objective? I do not think we are even talking 
the same terms when we talk evaluation. The question I wrote down this morning is: 

"What Is evaluation?" i'll bet we could not all write d<"wn a definition that was even 
similar. I question whether we could come out of this meeting today and put on that 
board a definition of evaluation that we aH can agree to. If we cannot do thuL, what 
have we done for the last two days? I ask you, "what is evaluation?" I do not know. 

1 am an evaluator, but I do not even know what it is. 

CHAIRMAN RISING; You have proved that you have done it? 

MR. JENKINS; No. I have not. You have accepted that I have proved I have done 
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CHAIRMAN! RISING: I didn’t accept that, and there are some others that have expres- 

sed doub' j , 

MRSr SHORT: I would like to mai:e a brief comment on Dr. Irwin Brownes comment. 

Our main activity for the last severel months has Leen to ask the physicians to partici- 
pate in criticizing what we are doin;:, what are putting into the program, how we are 
presenting the program, whether we are taking ir where it is needed, and what would be 
of enough interest and inspiration to them, offered in the most helpful way. If we 
could get them to be as critical of us as we a^e of them, perhaps we could thrash it 
out and produce something more useful. 

DR. LEMON: 1 Just wanted to point out that our discussion group, as you know, took 

as its major point that we thought we ought to get the practicing physician involved in 
evaluation of continuing education as well as his participation. That is the same 
section that Keith Jenkins was in, so we are concerned about that. I believe ic is very 
imporcant; we thought it most important. 

The second point I want to make is in ny area — at least my experience — is 
much like Marilyn Short ^s. We are very anxious to get physicians to do Jtist that. I 
think we are making progress but it is an uphill struggle, and 1 have some sympathy for 
it because I have been in practice. The reason. Is not that they are not interested in 
continuing education or our program It is because they are overwhelmed with other 
things. It Is hard to get practicing physicians as involved as we would like. 

DR IRWIN BROJ^ : Here, at least, Clem Brown and I agree. For years at the State 

Medical Society meetings I iiave maintained a booth to ask the practicing physicians for 
their opinions and Inviting their comments aid criticisms. I am sure we all do something 
like that- That is part of your evaluation, but what 1 was lalking about in the last 
day and one-half — and I was not in the other group meetings and did not get that out 
of the group reports — but it. the last day and one-half we have talked about motivation 
and other behavioral changes in physicians, and to my knowledge no one has been saying 
anything about motivation and behavior chcnges in the educator-evaluator. 

As a physician, I wonder what I would do if someoie came to evaluate me if L knew 
he was an evaluator, if I knev his motivation, and if I knew what I was going co do to 
change his behavior. If I thought he was going to evaluate me with c good motivation, 
whatever that is, and it was going to change his behavior so that he would pul ori a 
better program for me, I would donate my time. To my knowledge this has not been part 
of our subject material as a motivation behind what we are doing. You know, good patient 
care, everyone can get behind that screen and say, ’’good patient care." As a physician 
what concerns me is, ’’why is this evaluator motivated to spend his time checking into 
me-”’ 



DR. PERLMAN: From what Ir' ' vn said before, I think I was slightly misunderstood 

because we are equating two things which are not necessarily equatable. Dr. Jerry 
Royer will have to defend himself, but I am defending myself because 1 think it is 
important for the understanding of the group here, not because it bothers me. 

As a physician I make the assumption that ^<3 percent antibiotics — and that datum 
is valid, and if anything it is under-recordad — ^5 percent of patiencs with appen- 
dectomleo is Just too many. There are not that many complications in patiencs that 
need antibiotics in a routine appendectoTiy , Across the country we have a problem with 
the use of antibiotics, 

1 do not Imply from that, and that 1 think is the misunderstanding, that there- 
fore physicians' motivations are bad. 1 am saying the practice is not as good as it 
should be. That does not mean the motivations are bad. It just means that we have a 
problem of how to help them use their motivation. I tliink physicians are motivated. 

If alwayu felt that most are. The question is: how do we get them to he able to 

t knowledge we can impart, and to practice what they admit is good medicine? 

oJ 
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CHAIRMAN RISING: The literature has baen full of the business of ''antibiotic 

umbrella," prophylactic antibiotics are bad. This is not new. It has been going on 
now for years. We have had programs on it; we have had teaching on it, we have had 
things in which doctors in a pre-test and post-test would show a change of knowledge 
if they are answering what you told them. The point is they do not entirely believe 
you. It is pretty hard to take a doctor who was practicing before penicillin and con- 
vince him that penicillin is bad. It is just very hard. I happen to have baen one of 
them. I practiced a good part of my life before we had antibiotics, and it took 
several proofs before I was willing to accept that, and a lot doctors apparently 
still do not. 

MR. JENKINS: Then what does that say to you as an educator? You are noc being 

effective because you have not changed the behavior. 

CHAIRMAN RISING: That is right. 

HR, JENKINS: Then you failed as an <»ducator, 

CHAIRMAN RISING: Amen, 

MR. JENKINS: That is what we are after here, I think, 

CHAIRMAN RISING: Why don’t they believe us? 

MR. JENKINS: Have you gone back to find out? Did you go ask those men why they 

do not? Don't they trust you? You hrve been around in this area a long time, I 

would assume from what I hear. They are paying your fees. 

CHAIRMAN RISING: We have actually talked with them and asked why they do not oe- 

lieve us, and they just say; "Well, you know, for five years I have not had any trouble 
with penicillin, antibiotics, etc. I have not had a case of pseuv^omembranous entero- 
colitis." Until they have one, they will not believe us, 

MR JENKINS: What does that cay to you? Does that not say you need another tech- 

nique . 

CHAIRMAN RISING: It sure does. 

DR. BANK: I would presume that those who ha »e partaken of the riotous activities 

on many campuses at some time or another had social studies or civics in high schools 

Are you then saying that the high school situation is no good? 

MR. JENKINS: I am saying that it has failed, absolutely, one hundred percent. 

I‘ll buy thatl 

CHAIRMAN RISING: I do not think anyone is going to argue that we do not fail and 

that we should not try to evaluate, 

^^R. JENKINS: Then we have to change it. That is what the public school has not 

done. It has not changed. 

DR. CLEMENT BROWN: I want to respond to the antibiotic thing because we did a study 

on the use of antibiotics in our hospital. I am going to mention this and discuss it a 
bit because I think it gives you a little bit of an Idea of how much I think needs to 
be done to really change physicians' behavior. 

Because of some other studies we are doing it seemed to us that a lot of the anti- 
biotics were being used inappropriately. We reviewed fifty consecutive uses of anti- 
biotics by members of our medical departmeiit and we found thet only 30 percent of the 
^ tics used, in other words, fifteen instances with antibiotics indicated, in the 
dose, route, duration used, by our criteria. Those were my criteria and 
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those of a general practitioner working with me, and then we brought soineone in from 
the medical schools to react to what we thought, Wa told the people in the medical 
department what we found, We asked them each individually to take a slip of paper and 
write down No. 1 and No. 2, Next to No- 1 put down what it is they thought we found; 
and next to No> 2 put down where it is each thought he wanted to be in terms of appro- 
priate use of antibiotics. 

They d^d this individually. We collected them all. The average of where they 
thought they were v?as a 35 percent appropriate usage; and where they said they wanted 
to be was a 75 percent appropriate usage > \^e then revealed that they were at tJte 30 

percent appropriate uses and we gave them some examples of what we felt were inappro- 
priate uses, and they were willing to accept them. 

That was a start, but still we did not know what we were dealing with, except a 
Lot of inappropriate uses of antibiotics that they agreed on, and a great cognitive dis- 
tance between where they were and where they wanted to be^ 

We then gave each member of the medical department an examination, which half of 
them flunked and half passed. The average score was 69 percent, which was less than 
the 75 percent they agreed to on a practice level, but it was more than rwice as good 
<js their actual practice level at 30 percent t In giving rtiis examination, and forty-two 
of the forty-four members of the department took it, we made a further diagnosis of what 
some of their problems were — lack of knowledge, lack of problem-solving skills, be- 
cause we constructed a problem-solving type of examinition. 

We then held a series of conferences on antibiotics, using the examination piimarily 
as the point of discussion. Now, just with respect to sore throats or management of 
pharyngitis, or whatever a doctor might call It when you give someone an antibiotic for 
a sore throat We spent an hour-and-a-half in discussion on two different occasion.', 
with members of the medical department. I will not go into details of ciie kinds of things 
we discussed and the kinds cf agreements ro which we aiiived 

To give one more example, sure, the physicians agree that we ought to get a culture 
on everyone's throat before we give him an antibiotic ^ou haven't any idea in the world 
by looking, and loads of studies have been done on this. You have ro h.ive a culcute. 

It is the only way to find out- They have said in these discussions: "Ves, that is fine 

if you are at the hospital, but we are in our offices and we do not have out culture 
media, we do not have this." 

In Just this one area we developed a program whereby the lab would pick up from 
their offices every day ar\> cultures. We began using these little culture tubes, vrhich 
they could artually put in .leir pocket -- It will stay for twenty-tour bouts — and 

they can drop it by the hospital the next day We reduced the cost because we have 

anticipated, and we have come up with a three times grpater use of throat cuituies In 
our hospital from the out-patient service. We proriised them ve wouid reduce the* cost 
for this, so that would not be much of a factor. We worked out some other kinds of ar- 
rangements, That is one little tiny area. 

We have re-studied the situation, the use of antibiotics, and over-all we now 
find a 60 percent appropriate usage of antibiotics. It took a lot of activity, but 
they are still not where they wan to be. In fact, before v*e to.ld them that we asked 
them again: "Where do you want to be’" They felt at this point they reallv thought they 
ought to be at the 80 percent level. Still, we have a substantial distance to go. Vk^lat 
ve are now agreeing to Is individualized small-group sessions, depending on the kind 
of problems we nave determined from our evaluation of patient care, sc lhat a gtoup of 
physicians who are having problems, partlcularlv with use of an antibiotic, will get a 
series of conferences or whatever it is that the learning experience haf.pens to be. 

Unless we get involved in doing this kind of thing to this extent, I do not think 
expect to see the kinds of changes that are really needed. You know, to have 




had a one-hour session or even three hours on the use of antibiotics on the basis oi 
the 30 percent appropriate usage that 5;e originally started with, I du not think it 
would have had much efface ^ 
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Unfortunately, we did not do a controlled series and find out what it w»ould have 
don^, if we bad nade the same kind of changes, but there are a lot of indications, For 
example, not a single dose of chloramphenicol was inappropriately ordered in six months 
after this conference, not a single use of the most populaily used antibiotic before the 
conference, it s!ial 1 be nameless, which is an antibiotic that the disease that it should 
be used to treat has not been invented yet. Not a single dose of rhat, so we sort of 
flguie that the series of conferences and discussions that we have had might have had 
something to do with this change. 1 will gran^ that we really do not know. It could 
have been just happenstance, you know^ it might have happened anyway if we had never 
done anything. 

In teems of the cost of this tr^^l program -- ray saiaty, the girl who works with 
me, the cost of PAS, paying physicians $50.00 per hour for auditing charts -- all these 
factors are a cost to our hospital, and we have gone from primary appendectomies — just 
to give you o couple of examples of changes we have achieved -- primary appendectomies 
going from a 51 percent rate of acute appendicitis to 81 percent rate acute appendicitis 
on primary appendectomies, which has held up over two years; going from. 23 percent to a 
13 percent complication rate in hysterectomies; and 30 percent to 60 percent appropriate 
use of antibiotics already mentioned; from one subtotal hysterectomy a month to one 
subtotal hysterectomy a year. The total cost of this program is $45,000 a year at our 
hospital, and less- With a hospital with a $6,000,000 budget, somehov? It seems to me 
that this IS not an inordinate cost for evaluation, the whole process, evaluatjon, 
education, re-cycled back to evaluation- I do not think It is that expensive^ You 
can achieve change, but it takes what a number of people have been saying, particular- 
ly Keith Jenkins, it takes a lot more work cn our part as educators, a little ingenuit^’ 
a little inventiveness, and mostly just a lot of hard work. 

CHAIRMAN RISING: Dr Brown, do you think this is an indication tnac this kind of 

precision of behavioral analysis and intensity of education and degree of :ost-effect- 
Iveness is pretty uniquely appropriate in the community hospital as contrasted with a 
state institution that is-.-. , 

DR. CLEMENT BRULS: 1 don’t know, Jesse, T don’t know. 

DR BISH: Then the traditional organisation of continuing education programs that 

IS really in format a seminar, a wa -walk-of f type of thing probably does not lend 

Itself to the evaluation that you are suggesting? Is that right' Ihis kind of an 
evaluation is not suitable in that context? Would that be your Judgment' 

DR CLEliLKT BROl'JN : 1 don't know 1 d^in’t knew, I don’t work in that context. 

I don’t work like Jesse does here. 

DR- BISH: What I am getting at, such a program certainly does not have that 

continued relationship between those wh j carry on the program and follow-up, the inter- 
action, and some personal kind of dynamism Js brought to relate to the age of modicifi- 
cation. It surely does not have that- Then, the conclusion Is obvious, that we can't 
do that kind of an evaluation In our context? 

CHAIRMAN RISING: Not that cheaply. 

DR, CLE’lENT BROk'N : Probably not that cheaply, 1 don't know. I keep thinking there 

are some things that really make a difference th .t ve cov'd evaluate, even as people 
come from around Kansas to this center. ihere m’^st je some things we expect to learn 
when they come here and they go back and refer patients here, and these patients still 
have the same old diagnoses, they are not using any of the new diagnoses. They have 
not gotten any of the new drugs that Dr, Rising has told then about. There must be 

that you arc having a substantial Impact, or else that you ate 
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THOMAS BROl-JN: First, being the Brown here who is not a physician, and you 

have aske‘ for »^oiphysi cians to participate, X feel compelled to say something whether 
I have somethl*^ to say or not. 

CHAIRJIAN RISING: A lot of us do that. 

DRr THOflAS BRORT^; I think we have deteriorated to some extent here because we are 
getting off the prime topic, unless we are all defining what the prime topic is, Ve 
need to get a delineation of your model that works in a communtLy hospital under this 
circumstance, and we have to take a look at a delineation of something that works in a 
medical school In terns of evaluation, or something th >y are getting at Vanderbilt in 
terms of evaluation^ I believe that we have to delineate these, we have to communicate 
these, we have to analyze these and look for the appropriate usage, and wa have to 
generalize from this, 

I want to T.i> .. a plea for what Dr. Hugh Petersen is saying; I want to make a plea 
for what some of the others are saying, that we need to come together with these pro- 
cesses in some manner. We need to make it visible so that it can be replicated, so chat 
it can be tested to see what works under what kind of circumstances. I would like to 
simply get ny opinion in, that I think this is what we really need to addiess ourselves 

to r 



DR' JRUIK BROkH ; I wanted to ask Clem Brown one question before we go on, and 1 
am sure that you followed this through, but it seems so simple to say that the incidence 

of appendicitis went from 31 percent to 80 percent. It would also be interesting to 

know what percentage of ruptured appendices there were when you had 31 percent and 80 
percent* I do not know what it was, but I can give you an example: in the New England 

Journal of Medicine about 1950 there was an article on how one should make an absolute 

diagnosis of appendicitis in a child under the age of five. They bad 100 percent ac- 

cucacy on diagnoses, but they had four deaths in their small series. 

In the nospital 1 was In I became interested ir uiie subject, so 1 combed back 
through that age group, and there was about a 51 percent accuracy of diagnosis and 
zero mortality. There ace so many things that are complex that 1 would not want tne 
others here to think that a simple test of one thing, such as: '^Do you do a chroac 
culture?" Yes, I did e throat culture and I waited three days; therefore 1 got three 
days late in starting my treatment I did a throat culture and I started on anti- 

biotics with a 75 percent or 90 percent chance of being right on antibiotics kTiat 
happened no the culture after they got it? What was the meaning of it? Those are all 
very complex things You cannot simplify it. 

CHAIRMAN RISING: I recently heard a micro! ' ologi s t , an infectious disease person 

— 1 won^t say his name because I can't be sure — discussing this. He hazarded the 
guess that tii3 "inappropriate" use of penicillin i^as resp''v«=ible for the remarkabJe de- 
cline in the incidence of acute rheumatic fever that has been seen recently. This was 
something that he said could not be proved ol disproved, but it was an intriguing idea. 
Perhaps our best criteria are faulty! They often have been in the past. 

DR. CLEKEFT BROW < : I would like to respond to his question. In primary appendec*- 

tomies it went from 52 percent to 81 percent and stayed at 61 percent the last two yAars. 
We had fewer complications and we have had no deaths for the five years 1 have been 
there fron an appendectomy. We did not expect any. 

We cid thirty fewer appendectomies, which may be a little indication of sometl^ing. 
Almost all of our surg?ons do almost all of t' eir surgery in our hospital, so we know 
that they are not taking their appendectomies elsewhere. We are now doing thirty fever, 
we have fewer r.vjplica tions , but we know very precisely why the situation has changed, 
because of our eferral system. The greatest portion of the problem. \‘as that cur general 
practitioners and our interns were referring patients to surgery for an appendectomy, 
wore being operated on in less than six hours of hospital admission, I think 
latively rare that someone needs to have an appendectomy within six hours. Most 
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of the sucgeons chat 1 have a great regard for say that you can watr,h someone and very 
often things will change in that period of time and you will not need to operate, but 
if you were operating on them quick the signs would not change. 



DR. EISELE: I want to compliment Clem Brown on the very sophisticated studies they 

have been doing and the effectiveness of them, but this is not entirely new. Mora than 
twenty years ago I was involved in a little seventy-five bel hospital which did a medi- 
CaJ audit for the staff, and they immediately sec up their own medical standards com- 
mittee, twelve general practitioners on the staff at tills hospital, the only hospital in 
the coun:y, and they startf^d reviewing our own wock in a very unsophisticated manner. 
Three years later they invited me back. 1 found that the number of appendectomies was 
hair of what they had been doing over a five-yeat period before the study — before 
their committee started working. The number of tonsillectomies had fallen to one-half. 
Nonspecific pelvic surgery ha" dropped remarkably. 

in very unsophisticated simple ways they are reviewing their own work and, I chink, 
have effected very remarkable improvement in the quality of their work. They can be 
justly proud of this. 

I had to put this In as my testimony chat a medical staff in any hospital can 
organize itself to do its own evaluation of the quality of care and thereby educate the 
mambec s 

DR. KORST: I would like to change the subject for a little while — not that this 
is not vc-ry interesting. It is vety refreshing for me to attend a meeting like this 
w: h a number of educators, administrative people, a good mixture, but as a physician 
I would like to make a point here, that much of our discussion over tlie last two days 
has been directed at whar physicians car. do in evaluation of programs. One of the 
greatest problems facing the physician today is the overextension of his time, and 
I am sure you are all aware if you read the Saturday Review , every article you read 
Is the extension of the problems in manpower. Increasing students and physicians isn’t 
the answer to this. The answer to this is the answer to the question: ’'How are all of 

the other supports, the allied supports in medicine, going to be helpful In the future?" 
I do not think we have talked very much about how you, the non-M.D. people in this 
conference, figure into this- Are we realiy just sitting around a table telling the 
physician what more to do in an already extended situation? Physicians need help, the 
whole system needs help, and I would like to get some answers to that. 



HR JENKINS: I will respond as someone who has 

what I snould not be doing, 

Because he is not going to accept out judgment, 
others telling us how to do our job. 1 want to work 
want to sit down and 1 wane to throw out my ideas, I 
out with a plan of operation. I can come in and do 
table to you, but if I ask you to come in and do an 
your practice, you are immediately suspicious of me 
— until you know that I can be trusted net to hand 
that it might be useful, to my own benefit. Therefo 
as 1 see it, is to build trust. That means getting 
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certain things, once they are accep- 
judit of your hospital, let alone 
until you know me as an individual 
that information out co any pla:e 
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to me to be teachirg physicians and hospital ad'ninis tracors how to do an audit of their 
hospital so they can plan their continuing education program. This is what I Lhlnk we 
should be working toward, and 1 think it i? what we have been saying. 1 think the whole 
key to this is the topic on physician attitudes, because if we do not have the physician 
we have nothing! It is lead, because without you gentlemen — no matter how mich we may 
talk about you — without you we cannot do anything, we can never get anything done. 

MR. COCLE: I would like to comment as to what the educator can do for the physician 

in his practice. 1 am an educator rather than a physician. Firsts there are certain 
ways that we con help ycu improve your efficiency of practice, if given the opportunity 
of finding out what happens now in your practice. I think thece are techniques we can 
apply in terms of more efficient management, in terms of better record keeping. With 
respect to record keeping, I would like to add the idea that before we jump on the band- 
wagon of medical audit, let us realize that we may be evaluating the physician's record 
keeping rather than his patient care. This is the danger tliat we often overlook in the 
utilization of a medical audit. 

Beyond that, with the Academy I have several proposals under way at this time, one 
of them being a h.anpower utilization in pediatrics to include not only the utilization 
of physicians but the utiHzation of nurses and other allied hsalth workers- We need 
to consider the input from patients in this, and in pediatrics we certainly need to 
consider the parents. These are the kinds of things that educators can do to nelp phy- 
sicians improve the efficiency and effectiveness of their practice. Before ve can do 
this we have to be allowed to j'^cher some baseline data. This is the plea that we make 
to you ao educators: give us some baseline data to work with, and let us show you what 
might be possible, with your helpr 

DR. EISELE; Jesse, for twenty years I ) ive heard this criticism that we are eval- 
uating the physician's record keeping rather than his practice. Let me say, I cannot 
recall ever seeing an excellent record associated with bad practice, or vice versa. 

They generally go together pretty well. Pcor records are associated with poor practice, 
and the excellent records are associated with excellent practice. There is a very good 
correlation. 1 cannot give you a quotient, but this is ray observation. 

I have observed or detected some slight evidence of hostility. In fact, I think 
there have been a few outright doctor haters in the room. Let me tell you right now, 
if you are an educator or an evaluator and have this kind of an emotion, you better gee 
out of the business, '^ou are licked, 

HR. JENKINS: I don't think we are doctor haters. 

DR EISELE: I wasn't looking tt you, 

>[R. JENKINS: I know, but in the whole group, and I have talked with the others, 
and I do not think we are. We would not be here if we were, I hope not, and 1 reaJly 
do not believe so. It is a matter of if you hit head-on and you hit head-on, after 
a while you become frustrated. What do you start doing? You start bitching. Now, if 
we can stop hitting head-on, open the door and talk to each other, we may find out that 
both of us are pretty nice guys. 

DR. EISELE: My point is that the communications are just about zero when this 

type of attitude is,,., 

MR. JENKINS: All right, let as both make an attempt to open the door and not both 

be traveling the other way, 

DR. LESTER: 1 am a practicing physician and I came here to find out about this, 
by the way, 1 was told that everyone else was paid to come here, 1 was not. 

ERje 



I want to talk about this very thing. It is a concern to me, and ^as a concern 
before 1 c?me here, because I do think we should do a better job ot education 1 
think someone like nyself , or someone like Dr, William Fifer, who is a well-known 
practitioner who has now joined the RMP , can open doors that others cannot. 

I just want to say that these veiled threats of third-patty intervention and all 
that are fruitless. I seriously doubt that they can do the job if we can't. They’re 
not going to do it better because without our cooperation, as all of you have said, it 
cannot be done. I want you to know the whole effect of these little innuendoes on 
your future cooperation. This should not be part of your vocabulary. You should wipe 
the expression off your face. You should act as though you have heard of it but give 
it no credence, because we also read Saturday Rev i ew and all these other things. There 
is plenty that doctors can say about the motivation of other people -- electricians, 
carpenters, taxi drivers, and everybody else that provide services We all have feelings 
about senators and their efficiency, and about government and Its effect on inflation. 

The point being, however, and I hear fellow physicians say this: "What we really need 

to do by education is have some sort of a club." Academies have this and we tie member- 
ship to continuing effort, WTien I hear in the background, "you've got to do all this 
for these notable reasons, and if you do not, one day we are going to take your license," 
it has the same effect as the eighteen-year-old girl who comes in my office pregnant, 
who is not married, and who checks me out with tv;o or three visits to see whether she 
can trust me \cith information, or with herself, before she gets around to discussing 
her pregnancy They want to make sure that in the next room they cannot be heard when 
they discuss their private problems with me. If they v;ere perfect to start with, they 
would not have to see me as a physician. They do not need me. People who are imperfect 
need to come into my office, and their trust with what they tell me will be held in 
confidence is all important. 

From a practical standpoint, I have been waiting to hear, and it has not been 
mentioned: how do you assure, Dr. Brown, that the information you gather cannot be used 

by someone else? ’ 

DR, CLEMENT BROl^' : 1 do not know Mo one seems terclbly concerned, in our staff 

at least, 

CHAIRMAN RISING: For one reason, you have not published it. Most educators pub- 

lish their evaluations If they have been successful, at least- 

DR. CLEMENT BROWN: We do have an article coming out in JAMA , 

CHAIRMAN RISING: There may be a problem after it comes out 

DR. CLEMENT BROW’N: No, I don't think so, because 1 have had board ttuscee members 

and our staff members at Dr. Elsele’s meeting and we have talVrd about it ti some extent. 

Let me respond to the business of crust, which does seem to be a matter cf concern 
right at the moment. On this little opinion survey that was taken yesterday; "Most 
practicing physicians have little desire to keep up with the advances in medical 
practice." Only two people iiere agreed with that statement; ^5 disagreed with it. 

I think most people here sort of trust the motivations of a practicing physician, 
so 1 don’t think that is really too much of a problem. NTien [ first came 1 thought, 

"Oh boy.’ Here we go again; another two-day session of a bunch of educators and we ha'*e 
•*.o clearly stated objective," and the like kind of things that educators ate supposed 
CO say, I guess. Jesse, you stated the objective, and did you set my teeth on edge! 

Yc'i said: "The purpose of this conference is to write a book for — Health Manpower" 

— or something like that, "and no": to help those in attendance. We are all experts 
In what we are doing, and ve are here to sort ot write tnls report, book, or whatever," 
Like Wow! I don’t know why you get me so mad so quick at lines, Jesse. 

Q 1AIR>!AN RISING: I don’t know why you are so Irritable, Clem. 
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DR, CLEMENT BROWN: lhen> you know> we have been through this two days now» and I 

think back and I say> ’Vhat has happened?” We had some group discussions yesterday 
afternoon and the session this morning. Now> what we seem to be doing is diagnosing 
our needs as evaluators of continuing medical education. What are our problems, and 
how are we going to get arcund them? We have listed them under physician- learner 
problems, educator attitudes, costs, and measurement problems. 

Right now I am wondering, "have we done enough diagnosis tht:se last two days of 
what our problems are as potential CHE evaluators so the therapy is obvious to us, be- 
cause at least in our hospital we have often found that once you have performed the 
diagnosis the therapy is quite obvious. 

Here is where the real question comes in: "how will we know if our diagnosis has 

been sufficient and the therapy is obvious to ourselves as continuing medical education 
evaluators, because we have just be in through a two-day session as evaluators and we 
are not going to evaluate it at all?" 

I would propose that we evaluate what we have done the last two days by having 
Jesse Rising, if he would, mall out this opinion survey that everyone took the first 
dayv It would be interesting to see it there are some differences of opinion now. All 
of you have your original data. Even more important than that, if we have people not 
fill out these things on the bottom which really get at attitudes, but take No. 18: 

"I have developed a curricu\um for at least one of the CME programs I have conducted 
by first gathering data substantiating patient care deficits," I would like to know 
if 20 say "yes" or 30 say "yes" six months from now. 1 would like to know a year 
from now how many will say: "I have gathered data representing before and after, 

and I have conducted a program based on definite and significant improvement in patient 
care." I would like to see if we have, in fact, moved to doing anything as a result of 
this two-day session. 

Also, I would like to propose that we modify this and see if anything happens. 

Right now, fill out the top portion — and mail it back in — then, do a six-month or 
one-year follow-up because, we are talking about evaluation and we have not evaluated 
what we have done. We may need another session like this a year from now. We may have 
found out in that time that no one is doing any more evaluating than what we already 
have data on at this time, and we will have to realize that whatever happened today and 
yesterday did not he * o us to get ahead with the Job, and we need another session like 
this, but better, or different -- I do not know, 

CHAIRMAN RISING: Clem, why do you insist on changing my objective, I did not have 

an objective of educating this group, I told you that in the first place. This was 
not to vMucate you or to change your behavior, but to expose the ideas and thinking of 
those who are experts and practitioners of some phase of continuing education, 

DR. CLEMENT BROWN: That is what you told them when I got her ; I had some of my 

own objectives written cut, 

CHAIRMAN RISING: But those were objectives, 

DR. CLEMENT BROI^ : No, I shared them with my group yesterday, and they turned 

them all down, but that is fine. 

CHAIRMAN RISING: This was not planned as an educational exercise for us. I told 
you that, I made It perfectly clear in the letter I wrote. The objective was to get 
opinions of people who are "pros." Everyone here is presumably a professional in 
some phase of continuing education or evaluation. If that isn’t true, you came under 
false colors, because the letter did not go to anyone who was not. We wanted to get a 
book of opinions that can be put in the hards of others, I have already had people 
’ — they would like to use some of the things from this conference that will 
:ul to them if they can be quoted. So, don’t change my objective, please, Clem, 




DR, CLEMJiNT BRO^-TK: Suppose no one here is doing any more evaluating a year from 

now than what they are doing now? 

CHAIRM^J4 RISING: I really don^t care- That's their business — their choice. 

That has nothing to do with my objective. Some people may have been convinced for 
or even against by this... 

DR< CLEMNT BROWN: Most people that are here are not e^en doing any evaluating, 

by their own admission. How can we serve as experts to the rest of the world? 

CHAIRMAN RISING: I did not say they were all experts In evc.Iuaticn. 1 said they 
were all experts in some phase of continuing education. By your definition, of course, 
they are not doing continuing education if they do not do your kind of evaluation, which 
just possibly Is a little narrow-minded. 

Clem, I agree 100 percent with everything you said relative to your circumstances 
and your situation. I agree wholeheartedly with you that the over-all objective of 
continuing education Is to change behavior But the objective of the Minnesota Academy 
of G^^neral Practice’s program — even though it is partly fellowship and organizational 
maintenance -- the objective of their educational part of the program is rot necessarily 
an immediate change of behavior. It may be a third or fourth step back from that, by 
changing their members’ approach, by keeping their interest in education, and by stimu- 
lating their interest. This is one of the problems of applying Hugh Petersen’s criteria 
to absolutely everything until he can show us how we can measure change of attitude. 
Theoretically, it is possible, but the tools have not been here. The evaluations have 
not been evaluated. 

Please, let us not be paralyzed because we cannot evaluate everything in Hugh’s 
terms, Ii\ wcitlng a proposal for RMP It is really Important to write objectives that 
are easy to evaluate in terms of a change of behavior or attainment of the objectives, 
so that you have a proposal that will be suppot table by a renewal. Is thiw right, 
Thelma? 

MRS- SCHNEIDER: It would help, but it would also help to show that one day we 

might get around to patient care, Everything contributes toward that goal. 

CHAIRMAN RISING: This, of course, is another point. If vre ate talking about 

evaluating patient care, are we too late? It has been pointed out that we cannot 
eliminate mortality. People are going to die Ve can improve longevity very little 
unless we do away with degenerative diseases that no one, no matter how good an educa- 
tor, knows how to do. 

To improve patient care we are dealing with sick people because only sick people 
are patients. How do ve prevent Illness? Ideally, o one would be sick until the day 
he dies This is the goal wc are really aiming at in health care — the prevention of 
illness, the prevention of any sickness. 

If we have a pie-ln-the-sky approach to what out objectives must be we are going 
to have people say: "Well, phooey , It’s useless. I Just can’t do it." 

DR. I EMENT BROWN: Did you just say that we are not going to evaluate by finding 

out what people are doing six months or a year from now? 

CHAIRMAN RISING: I don't really rare. This particular group of people has al- 

ready heard all these arguments before, you know- You are not going to change their 
behavior dvatatically , because of hearing them once more. They probably are doing 
all they can. If they are not it is because they have not been convinced , Just like 
those who have been hearing about antibiotics. Unless we have control of this group 
In some way so we can control their behavior as you can In your hospital, we are not 
going to make the Minnesota Ac:.vlemy of General Practice do something that It cannot 
the name of common sense it can do. 
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DR. PERLMAN: Sure Jt cr.n. 



CHAlR^tAN RISING: How? 

DR. PERLMAN: He has availability of 23 percent of the patient discharges in 
Minnesota today. They have, not the details, but rough statistical data on 25 percent 
of the patients in Minnesota right now. 

CHAIRMAN RISINI: How is Dr. Lester going to say that the Academy of Gereral 

Practice has made fhls change by virtue of their program'? 

DR- PERLMAN: I do not think that is important. 

CHAItU^iriN RISING: Well, this Is what some appear to have been Saying. 

DR. PERLMAN: 1 said earlier that I would be satisfied — if we are going to try 
to prove things absolutely scientifically, 1 don't think we are ever going to get off 
the ground with an> of these programs — I think we are going tc have to be satisfied 
with relationships. We are going to have to be satisfied that L-dopa is a good drug 
if given in a higher, much greater amount and, hopefully, given properly a year from 
now- I don't think it matters, because this is such a fast-moving thing. 1 don't 
**hink it matters right now that we are going to have to try to give a percentage to 
what Joutnal adn versus an article in a journal versus a course that someone can, what 
percentage each was responsible for the ircreace. 

CHAIRMAN RISING: That's correct, Jack, but, you see, Dr- Fifer is not going to 

be able to take credit for this because It would have happened anyway. 

DR. PERLMAN; Why should he want credit? He says he does rot want to get credit. 

CHAIRMAN RISING: I know he doesn't want credit, but some of those here seem to 
think that the Minnesota Academy must have either credit or blame. 

DR. LEMON: The purpose of evaluation is to find out which method Is efiective. 

The Increase of L-dopa may be because of the increased activity of details and have 
nothing to do with Dr. Fifer 's program. 

DR. PERLMAN: That's right, and 1 think thecv'*. are some programs we cannot evaluate 

to get this definite proof 

CHAIRMAN RISING: Would you please repeat that? 

DR, PERLMAN: There are some of these programs that we ate not going to be able 

to evaluate because they move too fast. 

CHAIRMAN RISING; Is that any reason why we should not put them on? Are we "per- 
petrating a fraud?” 

DR. PERLMAN; I think you can take the same kind of program on some tiling that is 
not as fast-moving and as dramatic and see if it Joes make an effect- You can just 
apply it, you can Just take a "guesstimate,” an educated guess, of whether tills is going 
to have an effect. Is this some drug that Is so dramatic that it makes people so much 
better? 

DR. CLEMENT BROk^; You always do evaluate to see what kind of an effect you have 
had In continuing education. I would like to evaluate the use of L-dopa at some point 
a year or so from now to see if 70 percent of the people are using it inappropriately. 

I would then construct an educational program to get at the need If 90 percent of 
O 0 percent, whatever figure we want to accept, are using i? appropriately, let'e- 
it. Let's spend money on something else. 
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CHAIRMAN RISINic.: Everyone here Is getting a bit rescles 3 . The unanmicy that 

seemed to have developed in tae discussion groups yesterday afternoon is evaporating 
under the stress of fatigue, and flight departures are getting closer and closer. I 
will ask foi one final comment and I think we will then have to adjourn, 

MR, SCHMALGEMEIER: This is the first time 1 have been able to have the last 

word in a long time — my wife is at home. 

CHAIRMAN RISING: It ie now on the record. 

MR, SCHMALGEMEiER: We keep coming back to one thing, and that is the face tnat 

we have to establish the objectives for which we are carrying on our educational pro- 
gram. 



The educational program, or activity, in which we are involved here iu Kansas 
City has one objective, very easily evaluated; everyone here will be able to tick 
it off at the time it is completed when the book Is published. L think, Clem, that 
this is In a sense the basic behavioral objective of this course. I wish it were 
otherwise, but that is a different situation. It can easily be checked when it is 
published > 

1 think when we are talking about the over-all pattern of the educational evaluation 
we are talking about, we have to tie it down to the kind of objectives we have 1 wonder 
what Dr, Fifei’s objective is in offering a course on 3.-dopa at that time. If it is 
to Increase the usage of L-dopa he is going to be able to say; "Yes, the increase in 
usage In L-dopa came about " If he Is going to say It is to get Increased effective 
use of It, then he must do some evaluation of what the behavior v?as before and what the 
behavior is going to be afterwards. You cannot do it in a vacuum. If it is a head count 
you are looking for, forget th^ course, save the money, spend it on something else, have 
a good vacation, go to Bermuda — 1 do not care what :.t is -- the use will go on just 
the same. 

VTe are back to the same thing again that we talked about at the beginning of the 
program; that is, let us establish ou«^ basic objectives. Let us establish the kind cf 
evaluation techniques which will result from our objectives and then let us tesc the 
objectives to see whether we have carried them out 1 think that many of us, and I 
think that most of us, would say that those objectives would have to be In some kind 
of action rather than some passive result- Therefore, the behavioral type of cueput 
or objective is what we are looking for. We do want changes in processes and pro^^edures 
These things are available in math courses, such as we are carrying on in Michigan, and 
others are carrying on through the departments of postgraduate medicine They ace 
available through the hospital-oriented piograns, such as Clem Brown is working on, 

They are also available through seme of the other activities. 

May I put out one suggestion: if we cannot measure it, let^s not do it at all. 

CHAIRMAN RISING: Someone in the audience just said: "I hate to close on that 

note — if we cannot measure it, don't do it!" Tins makes the untenable assumption 
that, if a program isn’t evaluated, it Ips o facto is not educational. 

The last word will be that I want to thank the consultant planning group, Gail 
Bank, Clem Brown, Irvin Brown, Hugh Petersen, and Lee Stauffer (who, unfortunately, 
could not be with us) for helping us plan the format for this conference If It has 
not been succ*3sful, 1 will take the blame. If ar.y of you have enjoyed it, ot had any 
fun, which to me Is always an objective though not stated, or if there is anything that 
comes ou^^ in the book that is useful to anyone, It Is to their credit and to the credit 
of all of you. Thank you all for being here with us. It has been a real pleasure to 
host this conference -- it was not an eoucational exercise, it was a conference -- it 
has been a real pleasure to host it and to meet all of you. 1 have thoroughly enjoyed 
® I love a good argument- I apologize if I have talked too much, and 1 know 

CI\JC Thank you all again. 
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EVALUATING THE EVALUATORS 



Ethel Nurge, Ph.D. (Anthropology) 

It was n,t my privilege to attend the conference on evaluation in continuing medi- 
cal educaticu, Inasmuch at I was away from the Medical Center at the time that it was 
held. On ray return Dr. Jesse D. Rising told me about the conference and made available 
to me the program and the proceedings of the second day. I read both of these with a 
great deal of Interest and on Dr. Rising's suggestion have written the following as a 
commentary after, but I hope ralevant to, the synposlum. 

Postgraduate medical education is not new of course and yet the emergence of the 
field as a field is a recent phenomenon. Even more recent and marked is the acknowledge- 
ment of need for objective evaluation and a self-conscious and sophisticated search for 
evaluative techniques. In telling you this, I do not inform you. Perhaps what I can 
say, that will give some perspective and a new orientation, has to do with the nature of 
social change and man as a social animal. Man as an Innovator both creates and resists 
change, and both of those aspects, man creating and resisting change, are evident in the 
papers and discussion of the conference on evaluation in continuing medical education. 

When a novelty, innovation, or new behavior is urged on a group, their reaction to 
th" suggested change may be as wide as is the composition of the group, but the chances 
are that the responses will not be so wide-ranging. In fact, it may be helpful to think 
In terms of a series of selected responses, selected reactions to suggested change, i,e., 
the inauguration of evaluation as a standard and continuing procedure. It may be further 
helpful to think in terms of human beings as falling Into certain categories aa they are 
judged acceptors or rejectors of change. A wider understanding of man as an innovator 
or, conversely, staunch defendant of the status quo may help us to understand, predict, 
and effect social change. 



To discuss these who accept or reject an Idea we will begin by considering two in- 
terlocking hypotheses about the nature of change. The first Is that an individual will 
not. accept a novelty unless he is convinced that it satisfies a want better than some 
existing means at his disposal (Barnett 1953:378). 1 take this to wecin that the body of 
phyr.lcians as a whole will not welcome and support surveillance, evaluation, measurement 
of their behavior, performance, and practice unless they are clearly convinced that it 
does something for them that peer review or se] f-appraisal cannot do. I think that the 
participants of the present symposium have amply demonstrated that objective evaluation 
can valuably supplement peer review and self-appraisal. Indeed there are areas (which 
I leave to the experts to delineate) which only outside evaluation can effectively 
cover. Put the first hypothesis is that an individual will not accept a change unless 
he Is convinced that It satisfies a want better than some existing means at his disposal 



The second hypothesis is that there are biographical determinants antecedent to 
lack of satisfaction as a pervasive continuing state In a certain number of individuals. 
These individuals may have had specific and repeated disappointments which are a genesis 
for long-term pervasive dissatisfaction; it gives them a perspective which colors their 
views of large but variable sectors of their culture (Barnett 1953:319), It is these 
very individuals who, providing they are healthy, are most open to s*iggestlons for change. 
In other words, there are individuals who have predisposition to change and those who do 
not, and one lesson for u8 may be that we need to think of so training our medical 
students that they are less satisfied and more malleable and more open to change. 1 
think this touches on a peculiarly sensitive point in medical education since there is 
a traditional and, In many situations, a valid need for an assertive, authoritative pos- 
ture and character on the part of a physician. Such a posture, in most Instances, Is 
successfully mod^sled, copied, and well learned In medical school. To teach, at the same 
time, a contrary kind of behavior as a result of which the individual will not rest un- 
assumingly e his own interpretation, but rather constantly seek a reason and a way to 
it, presents a dilemma. In other words, pernaps have to find a way to make 
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the medical students dissatisfied with themselves at the same cime they are being taught 
to act as serene, knowledgeable characters. This Is neither easy to teach nor easy to 
practice. There is very little in the medical student's training or experience that 
makes him humble; there is a lot that makes him the contrary. 

At this point, let me enter a caveat. There are no clear-cut categories of indi- 
viduals who are acceptors as opposed to rejectors; no one person is wholly or persistent- 
ly an acceptor or rejector. He is simply more or less of one than the other = Still we 
may be able to make usef« 1 distinctions among the acceptors. Looking at those who are 
more commonly acceptors, ve find four categories. They are the dissident, the indif- 
ferent, the disaffected, and the resentful. 

Dissident . In every society there are individuals who have consistently refused 
to identify themselves with some of the norms and customs of their group. They may give 
lip service to avoid punishment but the more courageous and independent these dissenters 
are the more they openly rebel and withdraw from par ti t^ipation. Insofar as they with- 
draw from the expected participation we may look for alternate forms of behavior, that 
is, we may hope for acceptance of, and initiation of, change. 

The I nd iff erent . Many individuals are prepared to accept a new idea of behavior be- 
cause they have not dedicated themselves irretrievably to a custom or an ideal. In such 
individuals the receptive attitude is not due to dislike of an existing convention. In 
fact, rhey may be participants in the behavior to be changed, but they are not very en- i 

thuslastlc about the existing convention. They do not completely identify with the ideas 
and they do not get much satisfaction from them. These people are the indifferent and 
are a group whom we may expect to influence when seeking to change behavior. The great- 
est number of individuals in this category are young. Indeed the best example is aiJ- 
dren who are notably indifferent to the values and passions of their elders. For change 
targets among physicians, then, we should perhaps look to the youngest ones to find 
those who are Indifferent supporters of the s taLUS quo . 

Disaffected . Some people start out with a firm commitment to, and great satis- 
faction in, certain behaviors promulgated by their culture but then, somewhere along the 
line, they acquire a distaste for them. These are the disaffected and they too are a 
category of physicians whom we may seek out as being most vulnerable to, and accepting 
of, culture change. In propitious circumstances, presenting such individuals with al- 
ternative modes of behavior is enough to wean them away from their previous convictions. 

In our present circumstances that may very well mean individuals who have been first sat- 
isfied with medical care and then quite strongly critical of it are the persons most 
likely to be willing recipients of critics' charges for evaluation of continuing medical 
education and practice, and will, perhaps, take to heart Togo’s confession, "I have met 
the enemy and he is us '' 

Resentful , In every society there are those who are resentful and this is our 
fourth category of change target people. All societies have a limited number of the Cop 
pri^es, however such prizes may be defined, and always there are some who cannot get these 
prizes. One of the segments of a range of possible responses is resentment. These in- 
dividuals are not resigned to their fate and, by contrast with complacent Individuals 
of whom they may be envious, they are markedly receptive to suggestions for change. 

Therefore, when seeking targets for change behavior, seek among those who ate getting 
the least and wanting something better, maybe lower echelon physicians. 

Man as a social anitral Is no: unpredictable. Within limits, sometimes easily de- 
llmitable and sometimes not, we may speak of his propensities. In this paper I have 
taken as my problem a formulation of categories of potential acceptors of change. The 
specific change being desired is the acceptance of evaluation of medical education, not 
only by the medical educators — for they are fast moving to this themselves • — but al- 
so by other personnel. 
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